[==========] Running 56 tests from 4 test suites.
[----------] Global test environment set-up.
[----------] 30 tests from SecurityITest
[ RUN ] SecurityITest.TestAuthorizationOnListTablets
Loading random data
Initializing database '/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790561432251112-7085-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 02:10:32 dist-test-slave-86gv krb5kdc[7092](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 02:10:32 dist-test-slave-86gv krb5kdc[7092](info): set up 2 sockets
Sep 28 02:10:32 dist-test-slave-86gv krb5kdc[7092](info): commencing operation
krb5kdc: starting...
WARNING: Logging before InitGoogleLogging() is written to STDERR
W20260928 02:10:33.446060 7085 mini_kdc.cc:121] Time spent starting KDC: real 1.049s user 0.000s sys 0.016s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790561432251112-7085-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790561432251112-7085-0/krb5kdc/test-user.keytab.
Sep 28 02:10:33 dist-test-slave-86gv krb5kdc[7092](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561433, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T02:10:33Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T02:10:33Z Disabled control of system clock
WARNING: no policy specified for kudu/127.6.235.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.235.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.235.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790561432251112-7085-0/krb5kdc/kudu_127.6.235.126.keytab.
Entry for principal kudu/127.6.235.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790561432251112-7085-0/krb5kdc/kudu_127.6.235.126.keytab.
I20260928 02:10:33.641734 7085 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTlaUuc/build/asan/bin/kudu
/tmp/dist-test-taskTlaUuc/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790561432251112-7085-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790561432251112-7085-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790561432251112-7085-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790561432251112-7085-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.235.126:34429
--webserver_interface=127.6.235.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790561432251112-7085-0/krb5kdc/kudu_127.6.235.126.keytab
--principal=kudu/127.6.235.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.235.84:38563
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.235.126:34429
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790561432251112-7085-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790561432251112-7085-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790561432251112-7085-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:10:33.997538 7105 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:10:33.997860 7105 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:10:33.998025 7105 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:10:34.006285 7105 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 02:10:34.006398 7105 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:10:34.006497 7105 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:10:34.006603 7105 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 02:10:34.006690 7105 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 02:10:34.017107 7105 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.235.84:38563
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790561432251112-7085-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790561432251112-7085-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.235.126:34429
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790561432251112-7085-0/krb5kdc/kudu_127.6.235.126.keytab
--principal=kudu/127.6.235.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.235.126:34429
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790561432251112-7085-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.235.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790561432251112-7085-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 00:51:59 UTC on ccycloud.quasar-axgkwn.root.comops.site
build id 4676464
ASAN enabled
I20260928 02:10:34.018545 7105 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:10:34.019914 7105 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:10:34.037463 7110 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:10:34.037441 7111 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:10:34.037530 7113 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:10:34.038211 7105 server_base.cc:1034] running on GCE node
I20260928 02:10:34.039203 7105 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:10:34.040797 7105 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:10:34.042114 7105 hybrid_clock.cc:648] HybridClock initialized: now 1790561434042074 us; error 95 us; skew 500 ppm
I20260928 02:10:34.042557 7105 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 02:10:34 dist-test-slave-86gv krb5kdc[7092](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561434, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.235.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:10:34.046096 7105 init.cc:377] Logged in from keytab as kudu/127.6.235.126@KRBTEST.COM (short username kudu)
I20260928 02:10:34.047832 7105 webserver.cc:460] Webserver started at http://127.6.235.126:45967/ using document root <none> and password file <none>
I20260928 02:10:34.048604 7105 fs_manager.cc:359] Metadata directory not provided
I20260928 02:10:34.048758 7105 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:10:34.049134 7105 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:10:34.052454 7105 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790561432251112-7085-0/minicluster-data/master-0/data/instance:
uuid: "23398a375c62490b97c0ee25fca0842d"
format_stamp: "Formatted at 2026-09-28 02:10:34 on dist-test-slave-86gv"
server_key: "20954a36cb9d7844267a338054bd2799"
server_key_iv: "1974a479fddd0f41490992bbdae2ecae"
server_key_version: "encryptionkey@0"
I20260928 02:10:34.054647 7105 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790561432251112-7085-0/minicluster-data/master-0/wal/instance:
uuid: "23398a375c62490b97c0ee25fca0842d"
format_stamp: "Formatted at 2026-09-28 02:10:34 on dist-test-slave-86gv"
server_key: "20954a36cb9d7844267a338054bd2799"
server_key_iv: "1974a479fddd0f41490992bbdae2ecae"
server_key_version: "encryptionkey@0"
I20260928 02:10:34.060720 7105 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.001s
I20260928 02:10:34.065192 7120 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:10:34.066771 7105 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260928 02:10:34.066998 7105 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790561432251112-7085-0/minicluster-data/master-0/data,/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790561432251112-7085-0/minicluster-data/master-0/wal
uuid: "23398a375c62490b97c0ee25fca0842d"
format_stamp: "Formatted at 2026-09-28 02:10:34 on dist-test-slave-86gv"
server_key: "20954a36cb9d7844267a338054bd2799"
server_key_iv: "1974a479fddd0f41490992bbdae2ecae"
server_key_version: "encryptionkey@0"
I20260928 02:10:34.067220 7105 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790561432251112-7085-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790561432251112-7085-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790561432251112-7085-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:10:34.091972 7105 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:10:34.096290 7105 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:10:34.096611 7105 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:10:34.115811 7105 rpc_server.cc:307] RPC server started. Bound to: 127.6.235.126:34429
I20260928 02:10:34.115834 7171 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.235.126:34429 every 8 connection(s)
I20260928 02:10:34.117290 7105 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790561432251112-7085-0/minicluster-data/master-0/data/info.pb
I20260928 02:10:34.121224 7085 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTlaUuc/build/asan/bin/kudu as pid 7105
I20260928 02:10:34.121567 7085 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790561432251112-7085-0/minicluster-data/master-0/wal/instance
I20260928 02:10:34.121699 7172 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:10:34.124121 7085 external_mini_cluster.cc:1428] Setting key 0abf601ce1b7526e0c5019aa7e970db3
I20260928 02:10:34.137421 7172 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 23398a375c62490b97c0ee25fca0842d: Bootstrap starting.
I20260928 02:10:34.141597 7172 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 23398a375c62490b97c0ee25fca0842d: Neither blocks nor log segments found. Creating new log.
I20260928 02:10:34.142761 7172 log.cc:824] T 00000000000000000000000000000000 P 23398a375c62490b97c0ee25fca0842d: Log is configured to *not* fsync() on all Append() calls
I20260928 02:10:34.146286 7172 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 23398a375c62490b97c0ee25fca0842d: No bootstrap required, opened a new log
I20260928 02:10:34.154798 7172 raft_consensus.cc:348] T 00000000000000000000000000000000 P 23398a375c62490b97c0ee25fca0842d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "23398a375c62490b97c0ee25fca0842d" member_type: VOTER last_known_addr { host: "127.6.235.126" port: 34429 } }
I20260928 02:10:34.155287 7172 raft_consensus.cc:374] T 00000000000000000000000000000000 P 23398a375c62490b97c0ee25fca0842d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:10:34.155390 7172 raft_consensus.cc:729] T 00000000000000000000000000000000 P 23398a375c62490b97c0ee25fca0842d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 23398a375c62490b97c0ee25fca0842d, State: Initialized, Role: FOLLOWER
I20260928 02:10:34.156037 7172 consensus_queue.cc:260] T 00000000000000000000000000000000 P 23398a375c62490b97c0ee25fca0842d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "23398a375c62490b97c0ee25fca0842d" member_type: VOTER last_known_addr { host: "127.6.235.126" port: 34429 } }
I20260928 02:10:34.156243 7172 raft_consensus.cc:388] T 00000000000000000000000000000000 P 23398a375c62490b97c0ee25fca0842d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 02:10:34.156338 7172 raft_consensus.cc:482] T 00000000000000000000000000000000 P 23398a375c62490b97c0ee25fca0842d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 02:10:34.156466 7172 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 23398a375c62490b97c0ee25fca0842d [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:10:34.159420 7172 raft_consensus.cc:504] T 00000000000000000000000000000000 P 23398a375c62490b97c0ee25fca0842d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "23398a375c62490b97c0ee25fca0842d" member_type: VOTER last_known_addr { host: "127.6.235.126" port: 34429 } }
I20260928 02:10:34.160035 7172 leader_election.cc:302] T 00000000000000000000000000000000 P 23398a375c62490b97c0ee25fca0842d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 23398a375c62490b97c0ee25fca0842d; no voters:
I20260928 02:10:34.160737 7172 leader_election.cc:288] T 00000000000000000000000000000000 P 23398a375c62490b97c0ee25fca0842d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 02:10:34.160938 7177 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 23398a375c62490b97c0ee25fca0842d [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:10:34.161612 7177 raft_consensus.cc:686] T 00000000000000000000000000000000 P 23398a375c62490b97c0ee25fca0842d [term 1 LEADER]: Becoming Leader. State: Replica: 23398a375c62490b97c0ee25fca0842d, State: Running, Role: LEADER
I20260928 02:10:34.162176 7177 consensus_queue.cc:237] T 00000000000000000000000000000000 P 23398a375c62490b97c0ee25fca0842d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "23398a375c62490b97c0ee25fca0842d" member_type: VOTER last_known_addr { host: "127.6.235.126" port: 34429 } }
I20260928 02:10:34.162941 7172 sys_catalog.cc:564] T 00000000000000000000000000000000 P 23398a375c62490b97c0ee25fca0842d [sys.catalog]: configured and running, proceeding with master startup.
I20260928 02:10:34.165239 7179 sys_catalog.cc:455] T 00000000000000000000000000000000 P 23398a375c62490b97c0ee25fca0842d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 23398a375c62490b97c0ee25fca0842d. Latest consensus state: current_term: 1 leader_uuid: "23398a375c62490b97c0ee25fca0842d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "23398a375c62490b97c0ee25fca0842d" member_type: VOTER last_known_addr { host: "127.6.235.126" port: 34429 } } }
I20260928 02:10:34.165616 7179 sys_catalog.cc:458] T 00000000000000000000000000000000 P 23398a375c62490b97c0ee25fca0842d [sys.catalog]: This master's current role is: LEADER
I20260928 02:10:34.165407 7178 sys_catalog.cc:455] T 00000000000000000000000000000000 P 23398a375c62490b97c0ee25fca0842d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "23398a375c62490b97c0ee25fca0842d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "23398a375c62490b97c0ee25fca0842d" member_type: VOTER last_known_addr { host: "127.6.235.126" port: 34429 } } }
I20260928 02:10:34.166016 7178 sys_catalog.cc:458] T 00000000000000000000000000000000 P 23398a375c62490b97c0ee25fca0842d [sys.catalog]: This master's current role is: LEADER
I20260928 02:10:34.169893 7185 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
Sep 28 02:10:34 dist-test-slave-86gv krb5kdc[7092](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561433, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.235.126@KRBTEST.COM
I20260928 02:10:34.176827 7185 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 02:10:34.191263 7174 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:10:34.128994 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52590 (local address 127.6.235.126:34429)
0928 02:10:34.129933 (+ 939us) server_negotiation.cc:206] Beginning negotiation
0928 02:10:34.129958 (+ 25us) server_negotiation.cc:399] Waiting for connection header
0928 02:10:34.131225 (+ 1267us) server_negotiation.cc:407] Connection header received
0928 02:10:34.143726 (+ 12501us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:10:34.143757 (+ 31us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:10:34.144270 (+ 513us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:10:34.144607 (+ 337us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:10:34.148731 (+ 4124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:10:34.153493 (+ 4762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:10:34.157290 (+ 3797us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:10:34.157998 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:10:34.158198 (+ 200us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:10:34.177678 (+ 19480us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:10:34.177692 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:10:34.177722 (+ 30us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:10:34.177782 (+ 60us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:10:34.183417 (+ 5635us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:10:34.184712 (+ 1295us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:10:34.184722 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:10:34.184730 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:10:34.184966 (+ 236us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:10:34.185542 (+ 576us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:10:34.185547 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:10:34.185555 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:10:34.186318 (+ 763us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:10:34.186626 (+ 308us) server_negotiation.cc:1035] Waiting for connection context
0928 02:10:34.189441 (+ 2815us) server_negotiation.cc:299] Negotiation successful
0928 02:10:34.189875 (+ 434us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":632,"thread_start_us":386,"threads_started":1}
I20260928 02:10:34.191417 7185 catalog_manager.cc:1347] Generated new cluster ID: a7040001d33c484f973a33e3f51033c4
I20260928 02:10:34.192106 7185 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 02:10:34.227731 7185 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 02:10:34.229303 7185 catalog_manager.cc:1504] Loading token signing keys...
I20260928 02:10:34.242514 7185 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 23398a375c62490b97c0ee25fca0842d: Generated new TSK 0
I20260928 02:10:34.243680 7185 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.6.235.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.235.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.235.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790561432251112-7085-0/krb5kdc/kudu_127.6.235.65.keytab.
Entry for principal kudu/127.6.235.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790561432251112-7085-0/krb5kdc/kudu_127.6.235.65.keytab.
I20260928 02:10:34.304636 7085 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTlaUuc/build/asan/bin/kudu
/tmp/dist-test-taskTlaUuc/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790561432251112-7085-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790561432251112-7085-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790561432251112-7085-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790561432251112-7085-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.235.65:0
--local_ip_for_outbound_sockets=127.6.235.65
--webserver_interface=127.6.235.65
--webserver_port=0
--tserver_master_addrs=127.6.235.126:34429
--keytab_file=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790561432251112-7085-0/krb5kdc/kudu_127.6.235.65.keytab
--principal=kudu/127.6.235.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.235.84:38563
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790561432251112-7085-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790561432251112-7085-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790561432251112-7085-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:10:34.640295 7198 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:10:34.640590 7198 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:10:34.640687 7198 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:10:34.651500 7198 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:10:34.651640 7198 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:10:34.651853 7198 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.235.65
I20260928 02:10:34.663741 7198 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.235.84:38563
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790561432251112-7085-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790561432251112-7085-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790561432251112-7085-0/krb5kdc/kudu_127.6.235.65.keytab
--principal=kudu/127.6.235.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.235.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790561432251112-7085-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.6.235.65
--webserver_port=0
--tserver_master_addrs=127.6.235.126:34429
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.235.65
--log_dir=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790561432251112-7085-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 00:51:59 UTC on ccycloud.quasar-axgkwn.root.comops.site
build id 4676464
ASAN enabled
I20260928 02:10:34.665287 7198 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:10:34.666754 7198 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:10:34.681493 7207 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:10:34.681545 7204 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:10:34.681485 7203 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:10:34.682233 7198 server_base.cc:1034] running on GCE node
I20260928 02:10:34.682938 7198 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:10:34.684031 7198 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:10:34.685299 7198 hybrid_clock.cc:648] HybridClock initialized: now 1790561434685213 us; error 70 us; skew 500 ppm
I20260928 02:10:34.685693 7198 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 02:10:34 dist-test-slave-86gv krb5kdc[7092](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561434, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.235.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:10:34.689664 7198 init.cc:377] Logged in from keytab as kudu/127.6.235.65@KRBTEST.COM (short username kudu)
I20260928 02:10:34.691337 7198 webserver.cc:460] Webserver started at http://127.6.235.65:46555/ using document root <none> and password file <none>
I20260928 02:10:34.692215 7198 fs_manager.cc:359] Metadata directory not provided
I20260928 02:10:34.692368 7198 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:10:34.692780 7198 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:10:34.696187 7198 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790561432251112-7085-0/minicluster-data/ts-0/data/instance:
uuid: "5f61e0c8f7da4fb8bfc5a8200dc683ce"
format_stamp: "Formatted at 2026-09-28 02:10:34 on dist-test-slave-86gv"
server_key: "b5f71a04f0b47345251a7cf499d57d44"
server_key_iv: "3317ad2c65a67700a8cda9fa659994f1"
server_key_version: "encryptionkey@0"
I20260928 02:10:34.697083 7198 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790561432251112-7085-0/minicluster-data/ts-0/wal/instance:
uuid: "5f61e0c8f7da4fb8bfc5a8200dc683ce"
format_stamp: "Formatted at 2026-09-28 02:10:34 on dist-test-slave-86gv"
server_key: "b5f71a04f0b47345251a7cf499d57d44"
server_key_iv: "3317ad2c65a67700a8cda9fa659994f1"
server_key_version: "encryptionkey@0"
I20260928 02:10:34.702777 7198 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260928 02:10:34.706894 7214 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:10:34.708424 7198 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260928 02:10:34.708624 7198 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790561432251112-7085-0/minicluster-data/ts-0/data,/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790561432251112-7085-0/minicluster-data/ts-0/wal
uuid: "5f61e0c8f7da4fb8bfc5a8200dc683ce"
format_stamp: "Formatted at 2026-09-28 02:10:34 on dist-test-slave-86gv"
server_key: "b5f71a04f0b47345251a7cf499d57d44"
server_key_iv: "3317ad2c65a67700a8cda9fa659994f1"
server_key_version: "encryptionkey@0"
I20260928 02:10:34.708853 7198 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790561432251112-7085-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790561432251112-7085-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790561432251112-7085-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:10:34.730623 7198 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:10:34.735358 7198 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:10:34.735675 7198 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:10:34.736696 7198 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:10:34.738159 7198 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:10:34.738235 7198 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:10:34.738399 7198 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:10:34.738487 7198 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:10:34.772485 7198 rpc_server.cc:307] RPC server started. Bound to: 127.6.235.65:36517
I20260928 02:10:34.772523 7326 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.235.65:36517 every 8 connection(s)
I20260928 02:10:34.773882 7198 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790561432251112-7085-0/minicluster-data/ts-0/data/info.pb
I20260928 02:10:34.783612 7085 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTlaUuc/build/asan/bin/kudu as pid 7198
I20260928 02:10:34.783849 7085 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790561432251112-7085-0/minicluster-data/ts-0/wal/instance
I20260928 02:10:34.784570 7085 external_mini_cluster.cc:1428] Setting key 9fdd302eda9e596f0f3056deb3ff576e
Sep 28 02:10:34 dist-test-slave-86gv krb5kdc[7092](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561434, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.235.65@KRBTEST.COM for kudu/127.6.235.126@KRBTEST.COM
I20260928 02:10:34.796094 7329 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:10:34.776461 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.235.65:50575 (local address 127.6.235.126:34429)
0928 02:10:34.776862 (+ 401us) server_negotiation.cc:206] Beginning negotiation
0928 02:10:34.776867 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 02:10:34.778168 (+ 1301us) server_negotiation.cc:407] Connection header received
0928 02:10:34.780027 (+ 1859us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:10:34.780033 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:10:34.780145 (+ 112us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:10:34.780270 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:10:34.781903 (+ 1633us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:10:34.782597 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:10:34.784701 (+ 2104us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:10:34.785194 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:10:34.785340 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:10:34.789785 (+ 4445us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:10:34.789791 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:10:34.789794 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:10:34.789829 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:10:34.792618 (+ 2789us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:10:34.793647 (+ 1029us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:10:34.793651 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:10:34.793653 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:10:34.793776 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:10:34.794319 (+ 543us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:10:34.794322 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:10:34.794324 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:10:34.794586 (+ 262us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:10:34.794692 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0928 02:10:34.795669 (+ 977us) server_negotiation.cc:299] Negotiation successful
0928 02:10:34.795841 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":302,"thread_start_us":211,"threads_started":1}
I20260928 02:10:34.797801 7330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:10:34.777211 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.235.126:34429 (local address 127.6.235.65:50575)
0928 02:10:34.777999 (+ 788us) negotiation.cc:107] Waiting for socket to connect
0928 02:10:34.778043 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0928 02:10:34.779719 (+ 1676us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:10:34.780530 (+ 811us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:10:34.780539 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:10:34.781139 (+ 600us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:10:34.781728 (+ 589us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:10:34.781747 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:10:34.782761 (+ 1014us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:10:34.782771 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:10:34.784536 (+ 1765us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:10:34.784548 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:10:34.785444 (+ 896us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:10:34.785449 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:10:34.785768 (+ 319us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:10:34.786554 (+ 786us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:10:34.786592 (+ 38us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:10:34.789532 (+ 2940us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:10:34.792761 (+ 3229us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:10:34.792774 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:10:34.792797 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:10:34.793512 (+ 715us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:10:34.793880 (+ 368us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:10:34.793888 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:10:34.793895 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:10:34.794182 (+ 287us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:10:34.794707 (+ 525us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:10:34.794718 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:10:34.795216 (+ 498us) client_negotiation.cc:769] Sending connection context
0928 02:10:34.795557 (+ 341us) client_negotiation.cc:240] Negotiation successful
0928 02:10:34.796170 (+ 613us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":556,"thread_start_us":365,"threads_started":1}
I20260928 02:10:34.799403 7327 heartbeater.cc:347] Connected to a master server at 127.6.235.126:34429
I20260928 02:10:34.799860 7327 heartbeater.cc:464] Registering TS with master...
I20260928 02:10:34.800765 7327 heartbeater.cc:511] Master 127.6.235.126:34429 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.6.235.66@KRBTEST.COM; defaulting to no policy
I20260928 02:10:34.803102 7137 ts_manager.cc:194] Registered new tserver with Master: 5f61e0c8f7da4fb8bfc5a8200dc683ce (127.6.235.65:36517)
I20260928 02:10:34.804733 7137 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.235.65@KRBTEST.COM'} at 127.6.235.65:50575
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.235.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.235.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790561432251112-7085-0/krb5kdc/kudu_127.6.235.66.keytab.
Entry for principal kudu/127.6.235.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790561432251112-7085-0/krb5kdc/kudu_127.6.235.66.keytab.
I20260928 02:10:34.829618 7085 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTlaUuc/build/asan/bin/kudu
/tmp/dist-test-taskTlaUuc/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790561432251112-7085-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790561432251112-7085-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790561432251112-7085-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790561432251112-7085-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.235.66:0
--local_ip_for_outbound_sockets=127.6.235.66
--webserver_interface=127.6.235.66
--webserver_port=0
--tserver_master_addrs=127.6.235.126:34429
--keytab_file=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790561432251112-7085-0/krb5kdc/kudu_127.6.235.66.keytab
--principal=kudu/127.6.235.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.235.84:38563
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790561432251112-7085-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790561432251112-7085-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790561432251112-7085-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:10:35.171329 7333 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:10:35.171713 7333 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:10:35.171846 7333 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:10:35.179823 7333 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:10:35.179965 7333 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:10:35.180090 7333 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.235.66
I20260928 02:10:35.190446 7333 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.235.84:38563
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790561432251112-7085-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790561432251112-7085-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790561432251112-7085-0/krb5kdc/kudu_127.6.235.66.keytab
--principal=kudu/127.6.235.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.235.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790561432251112-7085-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.6.235.66
--webserver_port=0
--tserver_master_addrs=127.6.235.126:34429
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.235.66
--log_dir=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790561432251112-7085-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 00:51:59 UTC on ccycloud.quasar-axgkwn.root.comops.site
build id 4676464
ASAN enabled
I20260928 02:10:35.191773 7333 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:10:35.193391 7333 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:10:35.208069 7339 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:10:35.208134 7338 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:10:35.208442 7341 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:10:35.209261 7333 server_base.cc:1034] running on GCE node
I20260928 02:10:35.210078 7333 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:10:35.211151 7333 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:10:35.212373 7333 hybrid_clock.cc:648] HybridClock initialized: now 1790561435212338 us; error 35 us; skew 500 ppm
I20260928 02:10:35.212852 7333 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 02:10:35 dist-test-slave-86gv krb5kdc[7092](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561435, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.235.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:10:35.216207 7333 init.cc:377] Logged in from keytab as kudu/127.6.235.66@KRBTEST.COM (short username kudu)
I20260928 02:10:35.217875 7333 webserver.cc:460] Webserver started at http://127.6.235.66:33925/ using document root <none> and password file <none>
I20260928 02:10:35.218595 7333 fs_manager.cc:359] Metadata directory not provided
I20260928 02:10:35.218737 7333 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:10:35.219059 7333 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:10:35.222051 7333 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790561432251112-7085-0/minicluster-data/ts-1/data/instance:
uuid: "f2ad64a579b84acfbfbf574b04dcb4f7"
format_stamp: "Formatted at 2026-09-28 02:10:35 on dist-test-slave-86gv"
server_key: "8fc46e23c1df1b4e0537f82ad46aa72b"
server_key_iv: "e154db1de731c807bf830494723dc2d1"
server_key_version: "encryptionkey@0"
I20260928 02:10:35.222802 7333 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790561432251112-7085-0/minicluster-data/ts-1/wal/instance:
uuid: "f2ad64a579b84acfbfbf574b04dcb4f7"
format_stamp: "Formatted at 2026-09-28 02:10:35 on dist-test-slave-86gv"
server_key: "8fc46e23c1df1b4e0537f82ad46aa72b"
server_key_iv: "e154db1de731c807bf830494723dc2d1"
server_key_version: "encryptionkey@0"
I20260928 02:10:35.228276 7333 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260928 02:10:35.232187 7348 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:10:35.234007 7333 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 02:10:35.234191 7333 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790561432251112-7085-0/minicluster-data/ts-1/data,/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790561432251112-7085-0/minicluster-data/ts-1/wal
uuid: "f2ad64a579b84acfbfbf574b04dcb4f7"
format_stamp: "Formatted at 2026-09-28 02:10:35 on dist-test-slave-86gv"
server_key: "8fc46e23c1df1b4e0537f82ad46aa72b"
server_key_iv: "e154db1de731c807bf830494723dc2d1"
server_key_version: "encryptionkey@0"
I20260928 02:10:35.234400 7333 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790561432251112-7085-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790561432251112-7085-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790561432251112-7085-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:10:35.268743 7333 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:10:35.273504 7333 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:10:35.273799 7333 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:10:35.274816 7333 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:10:35.276381 7333 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:10:35.276482 7333 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:10:35.276635 7333 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:10:35.276710 7333 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:10:35.311072 7333 rpc_server.cc:307] RPC server started. Bound to: 127.6.235.66:37623
I20260928 02:10:35.311121 7460 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.235.66:37623 every 8 connection(s)
I20260928 02:10:35.312422 7333 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790561432251112-7085-0/minicluster-data/ts-1/data/info.pb
I20260928 02:10:35.319144 7085 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTlaUuc/build/asan/bin/kudu as pid 7333
I20260928 02:10:35.319350 7085 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790561432251112-7085-0/minicluster-data/ts-1/wal/instance
I20260928 02:10:35.319800 7085 external_mini_cluster.cc:1428] Setting key a5ee4409ebf531642f1dd200fe408d01
Sep 28 02:10:35 dist-test-slave-86gv krb5kdc[7092](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561435, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.235.66@KRBTEST.COM for kudu/127.6.235.126@KRBTEST.COM
I20260928 02:10:35.333959 7463 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:10:35.315213 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.235.66:49419 (local address 127.6.235.126:34429)
0928 02:10:35.315658 (+ 445us) server_negotiation.cc:206] Beginning negotiation
0928 02:10:35.315662 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:10:35.316622 (+ 960us) server_negotiation.cc:407] Connection header received
0928 02:10:35.318539 (+ 1917us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:10:35.318544 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:10:35.318640 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:10:35.318779 (+ 139us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:10:35.320485 (+ 1706us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:10:35.321497 (+ 1012us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:10:35.323378 (+ 1881us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:10:35.323813 (+ 435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:10:35.323979 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:10:35.328140 (+ 4161us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:10:35.328145 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:10:35.328149 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:10:35.328181 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:10:35.330805 (+ 2624us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:10:35.331578 (+ 773us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:10:35.331586 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:10:35.331590 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:10:35.331719 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:10:35.332147 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:10:35.332153 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:10:35.332156 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:10:35.332433 (+ 277us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:10:35.332585 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0928 02:10:35.333345 (+ 760us) server_negotiation.cc:299] Negotiation successful
0928 02:10:35.333560 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":352,"thread_start_us":242,"threads_started":1}
I20260928 02:10:35.335054 7464 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:10:35.315757 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.235.126:34429 (local address 127.6.235.66:49419)
0928 02:10:35.316448 (+ 691us) negotiation.cc:107] Waiting for socket to connect
0928 02:10:35.316488 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0928 02:10:35.318257 (+ 1769us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:10:35.319019 (+ 762us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:10:35.319028 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:10:35.319643 (+ 615us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:10:35.320273 (+ 630us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:10:35.320291 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:10:35.321687 (+ 1396us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:10:35.321712 (+ 25us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:10:35.323223 (+ 1511us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:10:35.323235 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:10:35.324085 (+ 850us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:10:35.324092 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:10:35.324320 (+ 228us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:10:35.325081 (+ 761us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:10:35.325107 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:10:35.327900 (+ 2793us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:10:35.330992 (+ 3092us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:10:35.331001 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:10:35.331019 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:10:35.331446 (+ 427us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:10:35.331832 (+ 386us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:10:35.331837 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:10:35.331841 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:10:35.332033 (+ 192us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:10:35.332557 (+ 524us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:10:35.332565 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:10:35.332985 (+ 420us) client_negotiation.cc:769] Sending connection context
0928 02:10:35.333279 (+ 294us) client_negotiation.cc:240] Negotiation successful
0928 02:10:35.333838 (+ 559us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":464,"thread_start_us":302,"threads_started":1}
WARNING: no policy specified for kudu/127.6.235.67@KRBTEST.COM; defaulting to no policy
I20260928 02:10:35.336757 7461 heartbeater.cc:347] Connected to a master server at 127.6.235.126:34429
I20260928 02:10:35.337098 7461 heartbeater.cc:464] Registering TS with master...
I20260928 02:10:35.337911 7461 heartbeater.cc:511] Master 127.6.235.126:34429 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.235.67@KRBTEST.COM" created.
I20260928 02:10:35.339458 7137 ts_manager.cc:194] Registered new tserver with Master: f2ad64a579b84acfbfbf574b04dcb4f7 (127.6.235.66:37623)
I20260928 02:10:35.340274 7137 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.235.66@KRBTEST.COM'} at 127.6.235.66:49419
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.235.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790561432251112-7085-0/krb5kdc/kudu_127.6.235.67.keytab.
Entry for principal kudu/127.6.235.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790561432251112-7085-0/krb5kdc/kudu_127.6.235.67.keytab.
I20260928 02:10:35.364457 7085 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTlaUuc/build/asan/bin/kudu
/tmp/dist-test-taskTlaUuc/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790561432251112-7085-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790561432251112-7085-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790561432251112-7085-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790561432251112-7085-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.235.67:0
--local_ip_for_outbound_sockets=127.6.235.67
--webserver_interface=127.6.235.67
--webserver_port=0
--tserver_master_addrs=127.6.235.126:34429
--keytab_file=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790561432251112-7085-0/krb5kdc/kudu_127.6.235.67.keytab
--principal=kudu/127.6.235.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.235.84:38563
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790561432251112-7085-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790561432251112-7085-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790561432251112-7085-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:10:35.698586 7467 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:10:35.698889 7467 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:10:35.698984 7467 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:10:35.706230 7467 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:10:35.706348 7467 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:10:35.706461 7467 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.235.67
I20260928 02:10:35.716388 7467 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.235.84:38563
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790561432251112-7085-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790561432251112-7085-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790561432251112-7085-0/krb5kdc/kudu_127.6.235.67.keytab
--principal=kudu/127.6.235.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.235.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790561432251112-7085-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.6.235.67
--webserver_port=0
--tserver_master_addrs=127.6.235.126:34429
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.235.67
--log_dir=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790561432251112-7085-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 00:51:59 UTC on ccycloud.quasar-axgkwn.root.comops.site
build id 4676464
ASAN enabled
I20260928 02:10:35.717681 7467 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:10:35.718997 7467 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:10:35.732648 7472 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:10:35.732685 7475 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:10:35.732719 7473 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:10:35.733618 7467 server_base.cc:1034] running on GCE node
I20260928 02:10:35.734278 7467 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:10:35.735231 7467 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:10:35.736493 7467 hybrid_clock.cc:648] HybridClock initialized: now 1790561435736418 us; error 69 us; skew 500 ppm
I20260928 02:10:35.736950 7467 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 02:10:35 dist-test-slave-86gv krb5kdc[7092](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561435, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.235.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:10:35.740236 7467 init.cc:377] Logged in from keytab as kudu/127.6.235.67@KRBTEST.COM (short username kudu)
I20260928 02:10:35.741761 7467 webserver.cc:460] Webserver started at http://127.6.235.67:36375/ using document root <none> and password file <none>
I20260928 02:10:35.742460 7467 fs_manager.cc:359] Metadata directory not provided
I20260928 02:10:35.742602 7467 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:10:35.742928 7467 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:10:35.746071 7467 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790561432251112-7085-0/minicluster-data/ts-2/data/instance:
uuid: "805a7a94bafc481a98ffe3830acfb87f"
format_stamp: "Formatted at 2026-09-28 02:10:35 on dist-test-slave-86gv"
server_key: "9a06532da29a013d42686cefcb4bc694"
server_key_iv: "08ca67e1e89f92471677626af2341823"
server_key_version: "encryptionkey@0"
I20260928 02:10:35.746968 7467 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790561432251112-7085-0/minicluster-data/ts-2/wal/instance:
uuid: "805a7a94bafc481a98ffe3830acfb87f"
format_stamp: "Formatted at 2026-09-28 02:10:35 on dist-test-slave-86gv"
server_key: "9a06532da29a013d42686cefcb4bc694"
server_key_iv: "08ca67e1e89f92471677626af2341823"
server_key_version: "encryptionkey@0"
I20260928 02:10:35.752746 7467 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20260928 02:10:35.756340 7482 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:10:35.757761 7467 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 02:10:35.757941 7467 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790561432251112-7085-0/minicluster-data/ts-2/data,/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790561432251112-7085-0/minicluster-data/ts-2/wal
uuid: "805a7a94bafc481a98ffe3830acfb87f"
format_stamp: "Formatted at 2026-09-28 02:10:35 on dist-test-slave-86gv"
server_key: "9a06532da29a013d42686cefcb4bc694"
server_key_iv: "08ca67e1e89f92471677626af2341823"
server_key_version: "encryptionkey@0"
I20260928 02:10:35.758140 7467 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790561432251112-7085-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790561432251112-7085-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790561432251112-7085-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:10:35.788460 7467 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:10:35.793329 7467 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:10:35.793628 7467 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:10:35.794600 7467 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:10:35.795929 7467 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:10:35.796033 7467 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:10:35.796190 7467 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:10:35.796271 7467 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:10:35.808017 7327 heartbeater.cc:503] Master 127.6.235.126:34429 was elected leader, sending a full tablet report...
I20260928 02:10:35.833762 7467 rpc_server.cc:307] RPC server started. Bound to: 127.6.235.67:46285
I20260928 02:10:35.833796 7594 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.235.67:46285 every 8 connection(s)
I20260928 02:10:35.835168 7467 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790561432251112-7085-0/minicluster-data/ts-2/data/info.pb
I20260928 02:10:35.843222 7085 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTlaUuc/build/asan/bin/kudu as pid 7467
I20260928 02:10:35.843433 7085 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790561432251112-7085-0/minicluster-data/ts-2/wal/instance
I20260928 02:10:35.843918 7085 external_mini_cluster.cc:1428] Setting key b02c790788b02b17684246c5e161ecbe
Sep 28 02:10:35 dist-test-slave-86gv krb5kdc[7092](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561435, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.235.67@KRBTEST.COM for kudu/127.6.235.126@KRBTEST.COM
I20260928 02:10:35.856884 7597 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:10:35.837987 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.235.67:46469 (local address 127.6.235.126:34429)
0928 02:10:35.838575 (+ 588us) server_negotiation.cc:206] Beginning negotiation
0928 02:10:35.838580 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 02:10:35.839572 (+ 992us) server_negotiation.cc:407] Connection header received
0928 02:10:35.841492 (+ 1920us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:10:35.841498 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:10:35.841595 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:10:35.841725 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:10:35.843569 (+ 1844us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:10:35.844376 (+ 807us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:10:35.846569 (+ 2193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:10:35.847007 (+ 438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:10:35.847150 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:10:35.850911 (+ 3761us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:10:35.850916 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:10:35.850919 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:10:35.850951 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:10:35.853673 (+ 2722us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:10:35.854509 (+ 836us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:10:35.854513 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:10:35.854515 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:10:35.854629 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:10:35.855072 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:10:35.855077 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:10:35.855081 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:10:35.855400 (+ 319us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:10:35.855565 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0928 02:10:35.856260 (+ 695us) server_negotiation.cc:299] Negotiation successful
0928 02:10:35.856484 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":433,"thread_start_us":277,"threads_started":1}
I20260928 02:10:35.857934 7598 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:10:35.838564 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.235.126:34429 (local address 127.6.235.67:46469)
0928 02:10:35.839307 (+ 743us) negotiation.cc:107] Waiting for socket to connect
0928 02:10:35.839429 (+ 122us) client_negotiation.cc:174] Beginning negotiation
0928 02:10:35.841201 (+ 1772us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:10:35.841968 (+ 767us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:10:35.841977 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:10:35.842508 (+ 531us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:10:35.843301 (+ 793us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:10:35.843329 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:10:35.844546 (+ 1217us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:10:35.844553 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:10:35.846418 (+ 1865us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:10:35.846431 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:10:35.847240 (+ 809us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:10:35.847248 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:10:35.847500 (+ 252us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:10:35.848187 (+ 687us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:10:35.848220 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:10:35.850680 (+ 2460us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:10:35.853850 (+ 3170us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:10:35.853858 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:10:35.853879 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:10:35.854337 (+ 458us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:10:35.854753 (+ 416us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:10:35.854762 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:10:35.854764 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:10:35.854952 (+ 188us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:10:35.855532 (+ 580us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:10:35.855540 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:10:35.855904 (+ 364us) client_negotiation.cc:769] Sending connection context
0928 02:10:35.856171 (+ 267us) client_negotiation.cc:240] Negotiation successful
0928 02:10:35.856749 (+ 578us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":491,"thread_start_us":306,"threads_started":1}
I20260928 02:10:35.859917 7595 heartbeater.cc:347] Connected to a master server at 127.6.235.126:34429
I20260928 02:10:35.860267 7595 heartbeater.cc:464] Registering TS with master...
I20260928 02:10:35.861181 7595 heartbeater.cc:511] Master 127.6.235.126:34429 requested a full tablet report, sending...
I20260928 02:10:35.863072 7136 ts_manager.cc:194] Registered new tserver with Master: 805a7a94bafc481a98ffe3830acfb87f (127.6.235.67:46285)
I20260928 02:10:35.864070 7136 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.235.67@KRBTEST.COM'} at 127.6.235.67:46469
I20260928 02:10:35.873245 7085 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 28 02:10:35 dist-test-slave-86gv krb5kdc[7092](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561435, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 28 02:10:35 dist-test-slave-86gv krb5kdc[7092](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561435, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.6.235.65@KRBTEST.COM
I20260928 02:10:35.920284 7603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:10:35.904216 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34066 (local address 127.6.235.65:36517)
0928 02:10:35.905017 (+ 801us) server_negotiation.cc:206] Beginning negotiation
0928 02:10:35.905025 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 02:10:35.905053 (+ 28us) server_negotiation.cc:407] Connection header received
0928 02:10:35.905121 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:10:35.905127 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:10:35.905348 (+ 221us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:10:35.905488 (+ 140us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:10:35.906912 (+ 1424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:10:35.907890 (+ 978us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:10:35.909283 (+ 1393us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:10:35.909841 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:10:35.909937 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:10:35.914499 (+ 4562us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:10:35.914510 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:10:35.914526 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:10:35.914561 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:10:35.917658 (+ 3097us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:10:35.918336 (+ 678us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:10:35.918343 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:10:35.918346 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:10:35.918551 (+ 205us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:10:35.918945 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:10:35.918955 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:10:35.918957 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:10:35.919303 (+ 346us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:10:35.919538 (+ 235us) server_negotiation.cc:1035] Waiting for connection context
0928 02:10:35.919790 (+ 252us) server_negotiation.cc:299] Negotiation successful
0928 02:10:35.920001 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":603,"thread_start_us":456,"threads_started":1}
W20260928 02:10:35.921764 7242 server_base.cc:1116] Unauthorized access attempt to method kudu.tserver.TabletServerService.ListTablets from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:34066
Sep 28 02:10:35 dist-test-slave-86gv krb5kdc[7092](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561435, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 28 02:10:35 dist-test-slave-86gv krb5kdc[7092](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561435, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.235.65@KRBTEST.COM
I20260928 02:10:35.967479 7603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:10:35.953691 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34072 (local address 127.6.235.65:36517)
0928 02:10:35.953878 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0928 02:10:35.953884 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 02:10:35.953987 (+ 103us) server_negotiation.cc:407] Connection header received
0928 02:10:35.954154 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:10:35.954158 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:10:35.954248 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:10:35.954332 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:10:35.955836 (+ 1504us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:10:35.956513 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:10:35.957807 (+ 1294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:10:35.958230 (+ 423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:10:35.958320 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:10:35.962248 (+ 3928us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:10:35.962253 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:10:35.962256 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:10:35.962294 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:10:35.965105 (+ 2811us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:10:35.965825 (+ 720us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:10:35.965829 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:10:35.965831 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:10:35.965950 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:10:35.966282 (+ 332us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:10:35.966285 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:10:35.966287 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:10:35.966559 (+ 272us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:10:35.966723 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0928 02:10:35.966986 (+ 263us) server_negotiation.cc:299] Negotiation successful
0928 02:10:35.967164 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260928 02:10:35.969229 7085 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskTlaUuc/build/asan/bin/kudu pid 7198
I20260928 02:10:35.982519 7603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:10:35.970286 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34076 (local address 127.6.235.65:36517)
0928 02:10:35.970478 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0928 02:10:35.970483 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 02:10:35.970953 (+ 470us) server_negotiation.cc:407] Connection header received
0928 02:10:35.971121 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:10:35.971125 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:10:35.971186 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:10:35.971298 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:10:35.972546 (+ 1248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:10:35.973344 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:10:35.974631 (+ 1287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:10:35.975104 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:10:35.975215 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:10:35.977503 (+ 2288us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:10:35.977508 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:10:35.977511 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:10:35.977545 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:10:35.980293 (+ 2748us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:10:35.980928 (+ 635us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:10:35.980935 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:10:35.980938 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:10:35.981071 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:10:35.981359 (+ 288us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:10:35.981362 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:10:35.981364 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:10:35.981621 (+ 257us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:10:35.981787 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0928 02:10:35.981999 (+ 212us) server_negotiation.cc:299] Negotiation successful
0928 02:10:35.982153 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260928 02:10:35.983197 7322 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 02:10:36.066221 7085 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTlaUuc/build/asan/bin/kudu with pid 7198
I20260928 02:10:36.083863 7085 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskTlaUuc/build/asan/bin/kudu pid 7333
Sep 28 02:10:36 dist-test-slave-86gv krb5kdc[7092](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561435, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.235.66@KRBTEST.COM
I20260928 02:10:36.100167 7610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:10:36.084856 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41928 (local address 127.6.235.66:37623)
0928 02:10:36.085448 (+ 592us) server_negotiation.cc:206] Beginning negotiation
0928 02:10:36.085456 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 02:10:36.085479 (+ 23us) server_negotiation.cc:407] Connection header received
0928 02:10:36.085541 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:10:36.085547 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:10:36.085757 (+ 210us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:10:36.085886 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:10:36.087123 (+ 1237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:10:36.088201 (+ 1078us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:10:36.089781 (+ 1580us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:10:36.090333 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:10:36.090435 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:10:36.094695 (+ 4260us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:10:36.094708 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:10:36.094723 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:10:36.094758 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:10:36.097543 (+ 2785us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:10:36.098215 (+ 672us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:10:36.098222 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:10:36.098225 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:10:36.098341 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:10:36.098692 (+ 351us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:10:36.098695 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:10:36.098697 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:10:36.099047 (+ 350us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:10:36.099270 (+ 223us) server_negotiation.cc:1035] Waiting for connection context
0928 02:10:36.099569 (+ 299us) server_negotiation.cc:299] Negotiation successful
0928 02:10:36.099763 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":386,"thread_start_us":248,"threads_started":1}
I20260928 02:10:36.101442 7456 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 02:10:36.185328 7085 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTlaUuc/build/asan/bin/kudu with pid 7333
I20260928 02:10:36.208607 7085 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskTlaUuc/build/asan/bin/kudu pid 7467
Sep 28 02:10:36 dist-test-slave-86gv krb5kdc[7092](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561435, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.235.67@KRBTEST.COM
I20260928 02:10:36.224968 7612 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:10:36.209622 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40170 (local address 127.6.235.67:46285)
0928 02:10:36.210228 (+ 606us) server_negotiation.cc:206] Beginning negotiation
0928 02:10:36.210235 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 02:10:36.210265 (+ 30us) server_negotiation.cc:407] Connection header received
0928 02:10:36.210327 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:10:36.210333 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:10:36.210545 (+ 212us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:10:36.210682 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:10:36.212002 (+ 1320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:10:36.213240 (+ 1238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:10:36.214561 (+ 1321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:10:36.215143 (+ 582us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:10:36.215246 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:10:36.219403 (+ 4157us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:10:36.219417 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:10:36.219433 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:10:36.219469 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:10:36.222301 (+ 2832us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:10:36.223044 (+ 743us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:10:36.223051 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:10:36.223053 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:10:36.223182 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:10:36.223566 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:10:36.223568 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:10:36.223570 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:10:36.223892 (+ 322us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:10:36.224150 (+ 258us) server_negotiation.cc:1035] Waiting for connection context
0928 02:10:36.224419 (+ 269us) server_negotiation.cc:299] Negotiation successful
0928 02:10:36.224627 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":423,"thread_start_us":272,"threads_started":1}
I20260928 02:10:36.226255 7590 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 02:10:36.315187 7085 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTlaUuc/build/asan/bin/kudu with pid 7467
I20260928 02:10:36.334678 7085 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskTlaUuc/build/asan/bin/kudu pid 7105
I20260928 02:10:36.336046 7167 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 02:10:36.439775 7085 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTlaUuc/build/asan/bin/kudu with pid 7105
2026-09-28T02:10:36Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnListTablets (4147 ms)
[ RUN ] SecurityITest.TestAuthorizationOnChecksum
Loading random data
Initializing database '/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790561432251112-7085-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 02:10:36 dist-test-slave-86gv krb5kdc[7617](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 02:10:36 dist-test-slave-86gv krb5kdc[7617](info): set up 2 sockets
Sep 28 02:10:36 dist-test-slave-86gv krb5kdc[7617](info): commencing operation
krb5kdc: starting...
W20260928 02:10:37.537386 7085 mini_kdc.cc:121] Time spent starting KDC: real 1.053s user 0.002s sys 0.014s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790561432251112-7085-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790561432251112-7085-0/krb5kdc/test-user.keytab.
Sep 28 02:10:37 dist-test-slave-86gv krb5kdc[7617](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561437, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T02:10:37Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T02:10:37Z Disabled control of system clock
WARNING: no policy specified for kudu/127.6.235.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.235.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.235.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790561432251112-7085-0/krb5kdc/kudu_127.6.235.126.keytab.
Entry for principal kudu/127.6.235.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790561432251112-7085-0/krb5kdc/kudu_127.6.235.126.keytab.
I20260928 02:10:37.729635 7085 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTlaUuc/build/asan/bin/kudu
/tmp/dist-test-taskTlaUuc/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790561432251112-7085-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790561432251112-7085-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790561432251112-7085-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790561432251112-7085-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.235.126:43611
--webserver_interface=127.6.235.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790561432251112-7085-0/krb5kdc/kudu_127.6.235.126.keytab
--principal=kudu/127.6.235.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.235.84:42021
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.235.126:43611
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790561432251112-7085-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790561432251112-7085-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790561432251112-7085-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:10:38.054946 7630 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:10:38.055245 7630 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:10:38.055408 7630 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:10:38.062708 7630 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 02:10:38.062822 7630 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:10:38.062915 7630 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:10:38.063017 7630 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 02:10:38.063097 7630 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 02:10:38.072969 7630 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.235.84:42021
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790561432251112-7085-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790561432251112-7085-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.235.126:43611
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790561432251112-7085-0/krb5kdc/kudu_127.6.235.126.keytab
--principal=kudu/127.6.235.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.235.126:43611
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790561432251112-7085-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.235.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790561432251112-7085-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 00:51:59 UTC on ccycloud.quasar-axgkwn.root.comops.site
build id 4676464
ASAN enabled
I20260928 02:10:38.074375 7630 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:10:38.075706 7630 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:10:38.087636 7635 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:10:38.087589 7638 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:10:38.087675 7636 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:10:38.088649 7630 server_base.cc:1034] running on GCE node
I20260928 02:10:38.089440 7630 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:10:38.090818 7630 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:10:38.092018 7630 hybrid_clock.cc:648] HybridClock initialized: now 1790561438091966 us; error 43 us; skew 500 ppm
I20260928 02:10:38.092317 7630 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 02:10:38 dist-test-slave-86gv krb5kdc[7617](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561438, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.235.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:10:38.095300 7630 init.cc:377] Logged in from keytab as kudu/127.6.235.126@KRBTEST.COM (short username kudu)
I20260928 02:10:38.096803 7630 webserver.cc:460] Webserver started at http://127.6.235.126:39787/ using document root <none> and password file <none>
I20260928 02:10:38.097529 7630 fs_manager.cc:359] Metadata directory not provided
I20260928 02:10:38.097637 7630 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:10:38.097893 7630 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:10:38.100732 7630 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790561432251112-7085-0/minicluster-data/master-0/data/instance:
uuid: "5ed0f422d11f429992a2b7a6a7e5331c"
format_stamp: "Formatted at 2026-09-28 02:10:38 on dist-test-slave-86gv"
server_key: "339bb72f40e7c05acb80143bfbd0b744"
server_key_iv: "3eb176248ff7e22f4c3e5d352e25f6d0"
server_key_version: "encryptionkey@0"
I20260928 02:10:38.101483 7630 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790561432251112-7085-0/minicluster-data/master-0/wal/instance:
uuid: "5ed0f422d11f429992a2b7a6a7e5331c"
format_stamp: "Formatted at 2026-09-28 02:10:38 on dist-test-slave-86gv"
server_key: "339bb72f40e7c05acb80143bfbd0b744"
server_key_iv: "3eb176248ff7e22f4c3e5d352e25f6d0"
server_key_version: "encryptionkey@0"
I20260928 02:10:38.106600 7630 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260928 02:10:38.110183 7645 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:10:38.111647 7630 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260928 02:10:38.111929 7630 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790561432251112-7085-0/minicluster-data/master-0/data,/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790561432251112-7085-0/minicluster-data/master-0/wal
uuid: "5ed0f422d11f429992a2b7a6a7e5331c"
format_stamp: "Formatted at 2026-09-28 02:10:38 on dist-test-slave-86gv"
server_key: "339bb72f40e7c05acb80143bfbd0b744"
server_key_iv: "3eb176248ff7e22f4c3e5d352e25f6d0"
server_key_version: "encryptionkey@0"
I20260928 02:10:38.112155 7630 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790561432251112-7085-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790561432251112-7085-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790561432251112-7085-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:10:38.133879 7630 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:10:38.138202 7630 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:10:38.138511 7630 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:10:38.157501 7630 rpc_server.cc:307] RPC server started. Bound to: 127.6.235.126:43611
I20260928 02:10:38.157550 7696 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.235.126:43611 every 8 connection(s)
I20260928 02:10:38.158888 7630 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790561432251112-7085-0/minicluster-data/master-0/data/info.pb
I20260928 02:10:38.162909 7697 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:10:38.168360 7085 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTlaUuc/build/asan/bin/kudu as pid 7630
I20260928 02:10:38.168628 7085 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790561432251112-7085-0/minicluster-data/master-0/wal/instance
I20260928 02:10:38.169545 7085 external_mini_cluster.cc:1428] Setting key 19b19d056acdea70e1aa3e11d1fa9d6e
I20260928 02:10:38.175731 7697 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5ed0f422d11f429992a2b7a6a7e5331c: Bootstrap starting.
I20260928 02:10:38.179375 7697 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5ed0f422d11f429992a2b7a6a7e5331c: Neither blocks nor log segments found. Creating new log.
I20260928 02:10:38.180577 7697 log.cc:824] T 00000000000000000000000000000000 P 5ed0f422d11f429992a2b7a6a7e5331c: Log is configured to *not* fsync() on all Append() calls
Sep 28 02:10:38 dist-test-slave-86gv krb5kdc[7617](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561437, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.235.126@KRBTEST.COM
I20260928 02:10:38.184062 7697 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5ed0f422d11f429992a2b7a6a7e5331c: No bootstrap required, opened a new log
I20260928 02:10:38.193358 7700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:10:38.171925 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58892 (local address 127.6.235.126:43611)
0928 02:10:38.172805 (+ 880us) server_negotiation.cc:206] Beginning negotiation
0928 02:10:38.172820 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0928 02:10:38.172866 (+ 46us) server_negotiation.cc:407] Connection header received
0928 02:10:38.174675 (+ 1809us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:10:38.174700 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:10:38.175123 (+ 423us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:10:38.175549 (+ 426us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:10:38.177069 (+ 1520us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:10:38.178432 (+ 1363us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:10:38.179769 (+ 1337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:10:38.180474 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:10:38.180691 (+ 217us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:10:38.185506 (+ 4815us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:10:38.185515 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:10:38.185531 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:10:38.185577 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:10:38.188684 (+ 3107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:10:38.189636 (+ 952us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:10:38.189644 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:10:38.189655 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:10:38.189841 (+ 186us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:10:38.190252 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:10:38.190255 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:10:38.190257 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:10:38.190829 (+ 572us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:10:38.191072 (+ 243us) server_negotiation.cc:1035] Waiting for connection context
0928 02:10:38.191657 (+ 585us) server_negotiation.cc:299] Negotiation successful
0928 02:10:38.192096 (+ 439us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":641,"thread_start_us":400,"threads_started":1}
I20260928 02:10:38.194311 7697 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5ed0f422d11f429992a2b7a6a7e5331c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5ed0f422d11f429992a2b7a6a7e5331c" member_type: VOTER last_known_addr { host: "127.6.235.126" port: 43611 } }
I20260928 02:10:38.194748 7697 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5ed0f422d11f429992a2b7a6a7e5331c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:10:38.194904 7697 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5ed0f422d11f429992a2b7a6a7e5331c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5ed0f422d11f429992a2b7a6a7e5331c, State: Initialized, Role: FOLLOWER
I20260928 02:10:38.195572 7697 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5ed0f422d11f429992a2b7a6a7e5331c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5ed0f422d11f429992a2b7a6a7e5331c" member_type: VOTER last_known_addr { host: "127.6.235.126" port: 43611 } }
I20260928 02:10:38.195865 7697 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5ed0f422d11f429992a2b7a6a7e5331c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 02:10:38.196033 7697 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5ed0f422d11f429992a2b7a6a7e5331c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 02:10:38.196229 7697 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5ed0f422d11f429992a2b7a6a7e5331c [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:10:38.198776 7697 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5ed0f422d11f429992a2b7a6a7e5331c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5ed0f422d11f429992a2b7a6a7e5331c" member_type: VOTER last_known_addr { host: "127.6.235.126" port: 43611 } }
I20260928 02:10:38.199421 7697 leader_election.cc:302] T 00000000000000000000000000000000 P 5ed0f422d11f429992a2b7a6a7e5331c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5ed0f422d11f429992a2b7a6a7e5331c; no voters:
I20260928 02:10:38.200040 7697 leader_election.cc:288] T 00000000000000000000000000000000 P 5ed0f422d11f429992a2b7a6a7e5331c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 02:10:38.200173 7702 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5ed0f422d11f429992a2b7a6a7e5331c [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:10:38.200923 7702 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5ed0f422d11f429992a2b7a6a7e5331c [term 1 LEADER]: Becoming Leader. State: Replica: 5ed0f422d11f429992a2b7a6a7e5331c, State: Running, Role: LEADER
I20260928 02:10:38.201532 7702 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5ed0f422d11f429992a2b7a6a7e5331c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5ed0f422d11f429992a2b7a6a7e5331c" member_type: VOTER last_known_addr { host: "127.6.235.126" port: 43611 } }
I20260928 02:10:38.202626 7697 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5ed0f422d11f429992a2b7a6a7e5331c [sys.catalog]: configured and running, proceeding with master startup.
I20260928 02:10:38.205991 7704 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5ed0f422d11f429992a2b7a6a7e5331c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5ed0f422d11f429992a2b7a6a7e5331c. Latest consensus state: current_term: 1 leader_uuid: "5ed0f422d11f429992a2b7a6a7e5331c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5ed0f422d11f429992a2b7a6a7e5331c" member_type: VOTER last_known_addr { host: "127.6.235.126" port: 43611 } } }
I20260928 02:10:38.206368 7704 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5ed0f422d11f429992a2b7a6a7e5331c [sys.catalog]: This master's current role is: LEADER
I20260928 02:10:38.206120 7703 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5ed0f422d11f429992a2b7a6a7e5331c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5ed0f422d11f429992a2b7a6a7e5331c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5ed0f422d11f429992a2b7a6a7e5331c" member_type: VOTER last_known_addr { host: "127.6.235.126" port: 43611 } } }
I20260928 02:10:38.206627 7703 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5ed0f422d11f429992a2b7a6a7e5331c [sys.catalog]: This master's current role is: LEADER
I20260928 02:10:38.209862 7707 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 02:10:38.215413 7707 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 02:10:38.224220 7707 catalog_manager.cc:1347] Generated new cluster ID: 8e9058bff3a64892bf6950382f8a76a5
I20260928 02:10:38.224357 7707 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 02:10:38.250613 7707 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 02:10:38.252494 7707 catalog_manager.cc:1504] Loading token signing keys...
I20260928 02:10:38.263230 7707 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5ed0f422d11f429992a2b7a6a7e5331c: Generated new TSK 0
I20260928 02:10:38.264362 7707 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.6.235.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.235.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.235.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790561432251112-7085-0/krb5kdc/kudu_127.6.235.65.keytab.
Entry for principal kudu/127.6.235.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790561432251112-7085-0/krb5kdc/kudu_127.6.235.65.keytab.
I20260928 02:10:38.317229 7085 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTlaUuc/build/asan/bin/kudu
/tmp/dist-test-taskTlaUuc/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790561432251112-7085-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790561432251112-7085-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790561432251112-7085-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790561432251112-7085-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.235.65:0
--local_ip_for_outbound_sockets=127.6.235.65
--webserver_interface=127.6.235.65
--webserver_port=0
--tserver_master_addrs=127.6.235.126:43611
--keytab_file=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790561432251112-7085-0/krb5kdc/kudu_127.6.235.65.keytab
--principal=kudu/127.6.235.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.235.84:42021
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790561432251112-7085-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790561432251112-7085-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790561432251112-7085-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:10:38.655966 7723 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:10:38.656256 7723 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:10:38.656360 7723 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:10:38.663825 7723 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:10:38.663946 7723 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:10:38.664072 7723 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.235.65
I20260928 02:10:38.674417 7723 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.235.84:42021
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790561432251112-7085-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790561432251112-7085-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790561432251112-7085-0/krb5kdc/kudu_127.6.235.65.keytab
--principal=kudu/127.6.235.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.235.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790561432251112-7085-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.6.235.65
--webserver_port=0
--tserver_master_addrs=127.6.235.126:43611
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.235.65
--log_dir=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790561432251112-7085-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 00:51:59 UTC on ccycloud.quasar-axgkwn.root.comops.site
build id 4676464
ASAN enabled
I20260928 02:10:38.675884 7723 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:10:38.677309 7723 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:10:38.690738 7729 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:10:38.690814 7731 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:10:38.690847 7728 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:10:38.691469 7723 server_base.cc:1034] running on GCE node
I20260928 02:10:38.692097 7723 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:10:38.693188 7723 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:10:38.694409 7723 hybrid_clock.cc:648] HybridClock initialized: now 1790561438694371 us; error 56 us; skew 500 ppm
I20260928 02:10:38.694738 7723 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 02:10:38 dist-test-slave-86gv krb5kdc[7617](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561438, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.235.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:10:38.697861 7723 init.cc:377] Logged in from keytab as kudu/127.6.235.65@KRBTEST.COM (short username kudu)
I20260928 02:10:38.699286 7723 webserver.cc:460] Webserver started at http://127.6.235.65:45633/ using document root <none> and password file <none>
I20260928 02:10:38.699980 7723 fs_manager.cc:359] Metadata directory not provided
I20260928 02:10:38.700124 7723 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:10:38.700433 7723 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:10:38.703531 7723 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790561432251112-7085-0/minicluster-data/ts-0/data/instance:
uuid: "d7bf36f0edb54e33973d0c5f7c82f3ad"
format_stamp: "Formatted at 2026-09-28 02:10:38 on dist-test-slave-86gv"
server_key: "15cd40975bf2fcda3905eb11e99ebff8"
server_key_iv: "52ef10dc14321b28d72be3171ddff484"
server_key_version: "encryptionkey@0"
I20260928 02:10:38.704398 7723 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790561432251112-7085-0/minicluster-data/ts-0/wal/instance:
uuid: "d7bf36f0edb54e33973d0c5f7c82f3ad"
format_stamp: "Formatted at 2026-09-28 02:10:38 on dist-test-slave-86gv"
server_key: "15cd40975bf2fcda3905eb11e99ebff8"
server_key_iv: "52ef10dc14321b28d72be3171ddff484"
server_key_version: "encryptionkey@0"
I20260928 02:10:38.709923 7723 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.002s
I20260928 02:10:38.713389 7738 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:10:38.714685 7723 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260928 02:10:38.714854 7723 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790561432251112-7085-0/minicluster-data/ts-0/data,/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790561432251112-7085-0/minicluster-data/ts-0/wal
uuid: "d7bf36f0edb54e33973d0c5f7c82f3ad"
format_stamp: "Formatted at 2026-09-28 02:10:38 on dist-test-slave-86gv"
server_key: "15cd40975bf2fcda3905eb11e99ebff8"
server_key_iv: "52ef10dc14321b28d72be3171ddff484"
server_key_version: "encryptionkey@0"
I20260928 02:10:38.715044 7723 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790561432251112-7085-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790561432251112-7085-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790561432251112-7085-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:10:38.730507 7723 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:10:38.734683 7723 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:10:38.734982 7723 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:10:38.735975 7723 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:10:38.737411 7723 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:10:38.737515 7723 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:10:38.737679 7723 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:10:38.737756 7723 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:10:38.770946 7723 rpc_server.cc:307] RPC server started. Bound to: 127.6.235.65:42579
I20260928 02:10:38.771034 7850 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.235.65:42579 every 8 connection(s)
I20260928 02:10:38.772274 7723 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790561432251112-7085-0/minicluster-data/ts-0/data/info.pb
I20260928 02:10:38.777338 7085 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTlaUuc/build/asan/bin/kudu as pid 7723
I20260928 02:10:38.777585 7085 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790561432251112-7085-0/minicluster-data/ts-0/wal/instance
I20260928 02:10:38.778185 7085 external_mini_cluster.cc:1428] Setting key 3fe76abd71d8d6f0132fc13bc3b495d2
Sep 28 02:10:38 dist-test-slave-86gv krb5kdc[7617](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561438, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.235.65@KRBTEST.COM for kudu/127.6.235.126@KRBTEST.COM
I20260928 02:10:38.795965 7853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:10:38.775074 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.235.65:35717 (local address 127.6.235.126:43611)
0928 02:10:38.775522 (+ 448us) server_negotiation.cc:206] Beginning negotiation
0928 02:10:38.775529 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 02:10:38.776556 (+ 1027us) server_negotiation.cc:407] Connection header received
0928 02:10:38.778563 (+ 2007us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:10:38.778569 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:10:38.778653 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:10:38.778791 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:10:38.780752 (+ 1961us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:10:38.781974 (+ 1222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:10:38.783606 (+ 1632us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:10:38.784311 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:10:38.784662 (+ 351us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:10:38.788845 (+ 4183us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:10:38.788855 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:10:38.788863 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:10:38.788935 (+ 72us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:10:38.792440 (+ 3505us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:10:38.793347 (+ 907us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:10:38.793354 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:10:38.793359 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:10:38.793529 (+ 170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:10:38.793958 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:10:38.793966 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:10:38.793971 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:10:38.794375 (+ 404us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:10:38.794571 (+ 196us) server_negotiation.cc:1035] Waiting for connection context
0928 02:10:38.795264 (+ 693us) server_negotiation.cc:299] Negotiation successful
0928 02:10:38.795525 (+ 261us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":323,"thread_start_us":206,"threads_started":1}
I20260928 02:10:38.796751 7854 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:10:38.775696 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.235.126:43611 (local address 127.6.235.65:35717)
0928 02:10:38.776395 (+ 699us) negotiation.cc:107] Waiting for socket to connect
0928 02:10:38.776440 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0928 02:10:38.778248 (+ 1808us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:10:38.779058 (+ 810us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:10:38.779066 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:10:38.779809 (+ 743us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:10:38.780569 (+ 760us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:10:38.780588 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:10:38.782126 (+ 1538us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:10:38.782129 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:10:38.783444 (+ 1315us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:10:38.783461 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:10:38.784733 (+ 1272us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:10:38.784742 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:10:38.785134 (+ 392us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:10:38.785980 (+ 846us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:10:38.786003 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:10:38.788574 (+ 2571us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:10:38.792627 (+ 4053us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:10:38.792634 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:10:38.792650 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:10:38.793217 (+ 567us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:10:38.793647 (+ 430us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:10:38.793651 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:10:38.793653 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:10:38.793836 (+ 183us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:10:38.794501 (+ 665us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:10:38.794506 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:10:38.794850 (+ 344us) client_negotiation.cc:769] Sending connection context
0928 02:10:38.795100 (+ 250us) client_negotiation.cc:240] Negotiation successful
0928 02:10:38.795505 (+ 405us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":463,"thread_start_us":273,"threads_started":1}
WARNING: no policy specified for kudu/127.6.235.66@KRBTEST.COM; defaulting to no policy
I20260928 02:10:38.798493 7851 heartbeater.cc:347] Connected to a master server at 127.6.235.126:43611
I20260928 02:10:38.798781 7851 heartbeater.cc:464] Registering TS with master...
I20260928 02:10:38.799618 7851 heartbeater.cc:511] Master 127.6.235.126:43611 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.235.66@KRBTEST.COM" created.
I20260928 02:10:38.801975 7662 ts_manager.cc:194] Registered new tserver with Master: d7bf36f0edb54e33973d0c5f7c82f3ad (127.6.235.65:42579)
I20260928 02:10:38.803639 7662 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.235.65@KRBTEST.COM'} at 127.6.235.65:35717
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.235.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790561432251112-7085-0/krb5kdc/kudu_127.6.235.66.keytab.
Entry for principal kudu/127.6.235.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790561432251112-7085-0/krb5kdc/kudu_127.6.235.66.keytab.
I20260928 02:10:38.824466 7085 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTlaUuc/build/asan/bin/kudu
/tmp/dist-test-taskTlaUuc/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790561432251112-7085-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790561432251112-7085-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790561432251112-7085-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790561432251112-7085-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.235.66:0
--local_ip_for_outbound_sockets=127.6.235.66
--webserver_interface=127.6.235.66
--webserver_port=0
--tserver_master_addrs=127.6.235.126:43611
--keytab_file=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790561432251112-7085-0/krb5kdc/kudu_127.6.235.66.keytab
--principal=kudu/127.6.235.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.235.84:42021
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790561432251112-7085-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790561432251112-7085-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790561432251112-7085-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:10:39.155295 7857 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:10:39.155598 7857 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:10:39.155756 7857 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:10:39.163205 7857 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:10:39.163365 7857 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:10:39.163641 7857 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.235.66
I20260928 02:10:39.177657 7857 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.235.84:42021
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790561432251112-7085-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790561432251112-7085-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790561432251112-7085-0/krb5kdc/kudu_127.6.235.66.keytab
--principal=kudu/127.6.235.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.235.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790561432251112-7085-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.6.235.66
--webserver_port=0
--tserver_master_addrs=127.6.235.126:43611
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.235.66
--log_dir=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790561432251112-7085-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 00:51:59 UTC on ccycloud.quasar-axgkwn.root.comops.site
build id 4676464
ASAN enabled
I20260928 02:10:39.179050 7857 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:10:39.180490 7857 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:10:39.198302 7862 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:10:39.198328 7863 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:10:39.198863 7865 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:10:39.206538 7857 server_base.cc:1034] running on GCE node
I20260928 02:10:39.207396 7857 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:10:39.208881 7857 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:10:39.210085 7857 hybrid_clock.cc:648] HybridClock initialized: now 1790561439210017 us; error 64 us; skew 500 ppm
I20260928 02:10:39.210431 7857 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 02:10:39 dist-test-slave-86gv krb5kdc[7617](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561439, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.235.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:10:39.218978 7857 init.cc:377] Logged in from keytab as kudu/127.6.235.66@KRBTEST.COM (short username kudu)
I20260928 02:10:39.221020 7857 webserver.cc:460] Webserver started at http://127.6.235.66:39701/ using document root <none> and password file <none>
I20260928 02:10:39.221908 7857 fs_manager.cc:359] Metadata directory not provided
I20260928 02:10:39.222034 7857 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:10:39.222349 7857 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:10:39.225912 7857 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790561432251112-7085-0/minicluster-data/ts-1/data/instance:
uuid: "5c0ce3b24633497da7336aa4b455d5b1"
format_stamp: "Formatted at 2026-09-28 02:10:39 on dist-test-slave-86gv"
server_key: "9e668635d13352080a0a56161024d26d"
server_key_iv: "31b02acc55db9b534e3616f3aa6d4e03"
server_key_version: "encryptionkey@0"
I20260928 02:10:39.226917 7857 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790561432251112-7085-0/minicluster-data/ts-1/wal/instance:
uuid: "5c0ce3b24633497da7336aa4b455d5b1"
format_stamp: "Formatted at 2026-09-28 02:10:39 on dist-test-slave-86gv"
server_key: "9e668635d13352080a0a56161024d26d"
server_key_iv: "31b02acc55db9b534e3616f3aa6d4e03"
server_key_version: "encryptionkey@0"
I20260928 02:10:39.233358 7857 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.001s
I20260928 02:10:39.237393 7872 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:10:39.238822 7857 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260928 02:10:39.238971 7857 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790561432251112-7085-0/minicluster-data/ts-1/data,/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790561432251112-7085-0/minicluster-data/ts-1/wal
uuid: "5c0ce3b24633497da7336aa4b455d5b1"
format_stamp: "Formatted at 2026-09-28 02:10:39 on dist-test-slave-86gv"
server_key: "9e668635d13352080a0a56161024d26d"
server_key_iv: "31b02acc55db9b534e3616f3aa6d4e03"
server_key_version: "encryptionkey@0"
I20260928 02:10:39.239148 7857 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790561432251112-7085-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790561432251112-7085-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790561432251112-7085-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:10:39.261047 7857 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:10:39.265883 7857 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:10:39.266192 7857 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:10:39.267164 7857 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:10:39.268633 7857 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:10:39.268746 7857 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:10:39.268926 7857 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:10:39.269004 7857 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.001s sys 0.000s
I20260928 02:10:39.303211 7857 rpc_server.cc:307] RPC server started. Bound to: 127.6.235.66:42755
I20260928 02:10:39.303251 7984 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.235.66:42755 every 8 connection(s)
I20260928 02:10:39.304656 7857 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790561432251112-7085-0/minicluster-data/ts-1/data/info.pb
I20260928 02:10:39.313251 7085 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTlaUuc/build/asan/bin/kudu as pid 7857
I20260928 02:10:39.313460 7085 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790561432251112-7085-0/minicluster-data/ts-1/wal/instance
I20260928 02:10:39.313988 7085 external_mini_cluster.cc:1428] Setting key b44cac1ffb19782220207c3c3a0ef847
Sep 28 02:10:39 dist-test-slave-86gv krb5kdc[7617](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561439, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.235.66@KRBTEST.COM for kudu/127.6.235.126@KRBTEST.COM
I20260928 02:10:39.327117 7987 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:10:39.307246 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.235.66:45919 (local address 127.6.235.126:43611)
0928 02:10:39.307738 (+ 492us) server_negotiation.cc:206] Beginning negotiation
0928 02:10:39.307744 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 02:10:39.308736 (+ 992us) server_negotiation.cc:407] Connection header received
0928 02:10:39.310632 (+ 1896us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:10:39.310637 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:10:39.310702 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:10:39.310849 (+ 147us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:10:39.312377 (+ 1528us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:10:39.313169 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:10:39.315299 (+ 2130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:10:39.315819 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:10:39.315951 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:10:39.320530 (+ 4579us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:10:39.320536 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:10:39.320550 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:10:39.320580 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:10:39.323737 (+ 3157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:10:39.324466 (+ 729us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:10:39.324475 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:10:39.324482 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:10:39.324633 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:10:39.325098 (+ 465us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:10:39.325106 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:10:39.325113 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:10:39.325494 (+ 381us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:10:39.325651 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0928 02:10:39.326388 (+ 737us) server_negotiation.cc:299] Negotiation successful
0928 02:10:39.326639 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":375,"thread_start_us":274,"threads_started":1}
I20260928 02:10:39.327814 7988 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:10:39.307864 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.235.126:43611 (local address 127.6.235.66:45919)
0928 02:10:39.308570 (+ 706us) negotiation.cc:107] Waiting for socket to connect
0928 02:10:39.308612 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0928 02:10:39.310363 (+ 1751us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:10:39.311106 (+ 743us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:10:39.311115 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:10:39.311622 (+ 507us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:10:39.312226 (+ 604us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:10:39.312249 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:10:39.313321 (+ 1072us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:10:39.313329 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:10:39.315152 (+ 1823us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:10:39.315167 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:10:39.316073 (+ 906us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:10:39.316080 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:10:39.316358 (+ 278us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:10:39.317246 (+ 888us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:10:39.317281 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:10:39.320292 (+ 3011us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:10:39.323891 (+ 3599us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:10:39.323899 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:10:39.323915 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:10:39.324344 (+ 429us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:10:39.324738 (+ 394us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:10:39.324741 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:10:39.324743 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:10:39.324988 (+ 245us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:10:39.325618 (+ 630us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:10:39.325624 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:10:39.325950 (+ 326us) client_negotiation.cc:769] Sending connection context
0928 02:10:39.326189 (+ 239us) client_negotiation.cc:240] Negotiation successful
0928 02:10:39.326645 (+ 456us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":467,"thread_start_us":269,"threads_started":1}
I20260928 02:10:39.329427 7985 heartbeater.cc:347] Connected to a master server at 127.6.235.126:43611
I20260928 02:10:39.329790 7985 heartbeater.cc:464] Registering TS with master...
I20260928 02:10:39.330509 7985 heartbeater.cc:511] Master 127.6.235.126:43611 requested a full tablet report, sending...
I20260928 02:10:39.332016 7662 ts_manager.cc:194] Registered new tserver with Master: 5c0ce3b24633497da7336aa4b455d5b1 (127.6.235.66:42755)
I20260928 02:10:39.333050 7662 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.235.66@KRBTEST.COM'} at 127.6.235.66:45919
WARNING: no policy specified for kudu/127.6.235.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.235.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.235.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790561432251112-7085-0/krb5kdc/kudu_127.6.235.67.keytab.
Entry for principal kudu/127.6.235.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790561432251112-7085-0/krb5kdc/kudu_127.6.235.67.keytab.
I20260928 02:10:39.361372 7085 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTlaUuc/build/asan/bin/kudu
/tmp/dist-test-taskTlaUuc/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790561432251112-7085-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790561432251112-7085-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790561432251112-7085-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790561432251112-7085-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.235.67:0
--local_ip_for_outbound_sockets=127.6.235.67
--webserver_interface=127.6.235.67
--webserver_port=0
--tserver_master_addrs=127.6.235.126:43611
--keytab_file=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790561432251112-7085-0/krb5kdc/kudu_127.6.235.67.keytab
--principal=kudu/127.6.235.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.235.84:42021
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790561432251112-7085-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790561432251112-7085-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790561432251112-7085-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:10:39.687397 7991 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:10:39.687678 7991 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:10:39.687834 7991 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:10:39.695194 7991 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:10:39.695312 7991 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:10:39.695484 7991 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.235.67
I20260928 02:10:39.705829 7991 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.235.84:42021
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790561432251112-7085-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790561432251112-7085-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790561432251112-7085-0/krb5kdc/kudu_127.6.235.67.keytab
--principal=kudu/127.6.235.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.235.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790561432251112-7085-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.6.235.67
--webserver_port=0
--tserver_master_addrs=127.6.235.126:43611
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.235.67
--log_dir=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790561432251112-7085-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 00:51:59 UTC on ccycloud.quasar-axgkwn.root.comops.site
build id 4676464
ASAN enabled
I20260928 02:10:39.707185 7991 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:10:39.708554 7991 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:10:39.721947 7996 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:10:39.722024 7999 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:10:39.722054 7997 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:10:39.722968 7991 server_base.cc:1034] running on GCE node
I20260928 02:10:39.723675 7991 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:10:39.724654 7991 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:10:39.725881 7991 hybrid_clock.cc:648] HybridClock initialized: now 1790561439725813 us; error 84 us; skew 500 ppm
I20260928 02:10:39.726226 7991 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 02:10:39 dist-test-slave-86gv krb5kdc[7617](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561439, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.235.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:10:39.729600 7991 init.cc:377] Logged in from keytab as kudu/127.6.235.67@KRBTEST.COM (short username kudu)
I20260928 02:10:39.731212 7991 webserver.cc:460] Webserver started at http://127.6.235.67:43183/ using document root <none> and password file <none>
I20260928 02:10:39.731987 7991 fs_manager.cc:359] Metadata directory not provided
I20260928 02:10:39.732152 7991 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:10:39.732502 7991 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:10:39.735912 7991 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790561432251112-7085-0/minicluster-data/ts-2/data/instance:
uuid: "db0e95f10adf4bb9aab02896f5cc77a0"
format_stamp: "Formatted at 2026-09-28 02:10:39 on dist-test-slave-86gv"
server_key: "1b3b234f219bb900cf52c7a19daacd99"
server_key_iv: "3ce5f93a6c908e89e25b095f8eaaca01"
server_key_version: "encryptionkey@0"
I20260928 02:10:39.736795 7991 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790561432251112-7085-0/minicluster-data/ts-2/wal/instance:
uuid: "db0e95f10adf4bb9aab02896f5cc77a0"
format_stamp: "Formatted at 2026-09-28 02:10:39 on dist-test-slave-86gv"
server_key: "1b3b234f219bb900cf52c7a19daacd99"
server_key_iv: "3ce5f93a6c908e89e25b095f8eaaca01"
server_key_version: "encryptionkey@0"
I20260928 02:10:39.742899 7991 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.003s sys 0.001s
I20260928 02:10:39.746968 8006 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:10:39.748689 7991 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260928 02:10:39.748831 7991 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790561432251112-7085-0/minicluster-data/ts-2/data,/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790561432251112-7085-0/minicluster-data/ts-2/wal
uuid: "db0e95f10adf4bb9aab02896f5cc77a0"
format_stamp: "Formatted at 2026-09-28 02:10:39 on dist-test-slave-86gv"
server_key: "1b3b234f219bb900cf52c7a19daacd99"
server_key_iv: "3ce5f93a6c908e89e25b095f8eaaca01"
server_key_version: "encryptionkey@0"
I20260928 02:10:39.749086 7991 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790561432251112-7085-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790561432251112-7085-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790561432251112-7085-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:10:39.777233 7991 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:10:39.782338 7991 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:10:39.782636 7991 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:10:39.783658 7991 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:10:39.785036 7991 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:10:39.785112 7991 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:10:39.785195 7991 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:10:39.785239 7991 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:10:39.806747 7851 heartbeater.cc:503] Master 127.6.235.126:43611 was elected leader, sending a full tablet report...
I20260928 02:10:39.819535 7991 rpc_server.cc:307] RPC server started. Bound to: 127.6.235.67:35003
I20260928 02:10:39.819619 8118 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.235.67:35003 every 8 connection(s)
I20260928 02:10:39.820998 7991 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790561432251112-7085-0/minicluster-data/ts-2/data/info.pb
I20260928 02:10:39.830117 7085 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTlaUuc/build/asan/bin/kudu as pid 7991
I20260928 02:10:39.830312 7085 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790561432251112-7085-0/minicluster-data/ts-2/wal/instance
I20260928 02:10:39.830835 7085 external_mini_cluster.cc:1428] Setting key 311109650bb1932ae578ed8bb780e7b3
Sep 28 02:10:39 dist-test-slave-86gv krb5kdc[7617](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561439, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.235.67@KRBTEST.COM for kudu/127.6.235.126@KRBTEST.COM
I20260928 02:10:39.841852 7987 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:10:39.823574 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.235.67:43891 (local address 127.6.235.126:43611)
0928 02:10:39.823770 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0928 02:10:39.823777 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 02:10:39.825091 (+ 1314us) server_negotiation.cc:407] Connection header received
0928 02:10:39.826849 (+ 1758us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:10:39.826853 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:10:39.826922 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:10:39.827011 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:10:39.828554 (+ 1543us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:10:39.829616 (+ 1062us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:10:39.831899 (+ 2283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:10:39.832451 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:10:39.832556 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:10:39.836700 (+ 4144us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:10:39.836705 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:10:39.836708 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:10:39.836741 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:10:39.839196 (+ 2455us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:10:39.839912 (+ 716us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:10:39.839917 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:10:39.839919 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:10:39.840027 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:10:39.840385 (+ 358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:10:39.840388 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:10:39.840390 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:10:39.840649 (+ 259us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:10:39.840761 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0928 02:10:39.841459 (+ 698us) server_negotiation.cc:299] Negotiation successful
0928 02:10:39.841633 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260928 02:10:39.843016 8121 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:10:39.824164 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.235.126:43611 (local address 127.6.235.67:43891)
0928 02:10:39.824866 (+ 702us) negotiation.cc:107] Waiting for socket to connect
0928 02:10:39.824976 (+ 110us) client_negotiation.cc:174] Beginning negotiation
0928 02:10:39.826596 (+ 1620us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:10:39.827218 (+ 622us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:10:39.827226 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:10:39.827773 (+ 547us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:10:39.828368 (+ 595us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:10:39.828386 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:10:39.829778 (+ 1392us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:10:39.829785 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:10:39.831726 (+ 1941us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:10:39.831740 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:10:39.832719 (+ 979us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:10:39.832727 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:10:39.833127 (+ 400us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:10:39.833964 (+ 837us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:10:39.833998 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:10:39.836472 (+ 2474us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:10:39.839324 (+ 2852us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:10:39.839331 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:10:39.839345 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:10:39.839809 (+ 464us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:10:39.840120 (+ 311us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:10:39.840124 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:10:39.840126 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:10:39.840295 (+ 169us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:10:39.840755 (+ 460us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:10:39.840761 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:10:39.841145 (+ 384us) client_negotiation.cc:769] Sending connection context
0928 02:10:39.841380 (+ 235us) client_negotiation.cc:240] Negotiation successful
0928 02:10:39.841767 (+ 387us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":475,"thread_start_us":270,"threads_started":1}
I20260928 02:10:39.844381 8119 heartbeater.cc:347] Connected to a master server at 127.6.235.126:43611
I20260928 02:10:39.844715 8119 heartbeater.cc:464] Registering TS with master...
I20260928 02:10:39.845664 8119 heartbeater.cc:511] Master 127.6.235.126:43611 requested a full tablet report, sending...
I20260928 02:10:39.847265 7662 ts_manager.cc:194] Registered new tserver with Master: db0e95f10adf4bb9aab02896f5cc77a0 (127.6.235.67:35003)
I20260928 02:10:39.848004 7662 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.235.67@KRBTEST.COM'} at 127.6.235.67:43891
I20260928 02:10:39.858861 7085 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 02:10:39.877351 7987 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:10:39.865268 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58894 (local address 127.6.235.126:43611)
0928 02:10:39.865484 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0928 02:10:39.865494 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 02:10:39.865926 (+ 432us) server_negotiation.cc:407] Connection header received
0928 02:10:39.866080 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:10:39.866084 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:10:39.866148 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:10:39.866229 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:10:39.867625 (+ 1396us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:10:39.868326 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:10:39.869631 (+ 1305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:10:39.870019 (+ 388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:10:39.870119 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:10:39.872411 (+ 2292us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:10:39.872416 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:10:39.872419 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:10:39.872449 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:10:39.874894 (+ 2445us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:10:39.875500 (+ 606us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:10:39.875504 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:10:39.875506 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:10:39.875630 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:10:39.875934 (+ 304us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:10:39.875937 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:10:39.875939 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:10:39.876233 (+ 294us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:10:39.876420 (+ 187us) server_negotiation.cc:1035] Waiting for connection context
0928 02:10:39.876749 (+ 329us) server_negotiation.cc:299] Negotiation successful
0928 02:10:39.876972 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260928 02:10:39.884608 7662 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58894:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 02:10:39.891539 7662 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 02:10:39.931670 8136 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:10:39.906504 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45844 (local address 127.6.235.66:42755)
0928 02:10:39.922003 (+ 15499us) server_negotiation.cc:206] Beginning negotiation
0928 02:10:39.922015 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0928 02:10:39.923488 (+ 1473us) server_negotiation.cc:407] Connection header received
0928 02:10:39.923841 (+ 353us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:10:39.923848 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:10:39.924076 (+ 228us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:10:39.924308 (+ 232us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:10:39.925103 (+ 795us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:10:39.926695 (+ 1592us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:10:39.929012 (+ 2317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:10:39.930242 (+ 1230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:10:39.930369 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:10:39.930604 (+ 235us) server_negotiation.cc:1035] Waiting for connection context
0928 02:10:39.931140 (+ 536us) server_negotiation.cc:299] Negotiation successful
0928 02:10:39.931306 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15332,"thread_start_us":15202,"threads_started":1}
I20260928 02:10:39.931708 8131 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:10:39.906349 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.235.66:42755 (local address 127.0.0.1:45844)
0928 02:10:39.923348 (+ 16999us) negotiation.cc:107] Waiting for socket to connect
0928 02:10:39.923372 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0928 02:10:39.923631 (+ 259us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:10:39.924341 (+ 710us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:10:39.924346 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:10:39.924370 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:10:39.924915 (+ 545us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:10:39.924931 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:10:39.926877 (+ 1946us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:10:39.926887 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:10:39.928800 (+ 1913us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:10:39.928813 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:10:39.930723 (+ 1910us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:10:39.930729 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:10:39.930902 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:10:39.930936 (+ 34us) client_negotiation.cc:769] Sending connection context
0928 02:10:39.931086 (+ 150us) client_negotiation.cc:240] Negotiation successful
0928 02:10:39.931283 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":16841,"mutex_wait_us":203,"thread_start_us":14636,"threads_started":1}
I20260928 02:10:39.933499 8134 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:10:39.905121 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48096 (local address 127.6.235.65:42579)
0928 02:10:39.922401 (+ 17280us) server_negotiation.cc:206] Beginning negotiation
0928 02:10:39.922413 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0928 02:10:39.922452 (+ 39us) server_negotiation.cc:407] Connection header received
0928 02:10:39.922596 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:10:39.922608 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:10:39.922960 (+ 352us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:10:39.923189 (+ 229us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:10:39.924151 (+ 962us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:10:39.926305 (+ 2154us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:10:39.929936 (+ 3631us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:10:39.931634 (+ 1698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:10:39.931788 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:10:39.932055 (+ 267us) server_negotiation.cc:1035] Waiting for connection context
0928 02:10:39.933001 (+ 946us) server_negotiation.cc:299] Negotiation successful
0928 02:10:39.933181 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17094,"thread_start_us":16945,"threads_started":1}
I20260928 02:10:39.933630 8135 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:10:39.904584 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.235.65:42579 (local address 127.0.0.1:48096)
0928 02:10:39.921314 (+ 16730us) negotiation.cc:107] Waiting for socket to connect
0928 02:10:39.921340 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0928 02:10:39.921490 (+ 150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:10:39.923491 (+ 2001us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:10:39.923499 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:10:39.923532 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:10:39.923946 (+ 414us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:10:39.923958 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:10:39.926607 (+ 2649us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:10:39.926617 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:10:39.929704 (+ 3087us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:10:39.929722 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:10:39.932243 (+ 2521us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:10:39.932253 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:10:39.932637 (+ 384us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:10:39.932802 (+ 165us) client_negotiation.cc:769] Sending connection context
0928 02:10:39.932939 (+ 137us) client_negotiation.cc:240] Negotiation successful
0928 02:10:39.933126 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":16551,"mutex_wait_us":174,"spinlock_wait_cycles":1920,"thread_start_us":10901,"threads_started":1}
I20260928 02:10:39.933733 8133 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:10:39.904666 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58308 (local address 127.6.235.67:35003)
0928 02:10:39.923717 (+ 19051us) server_negotiation.cc:206] Beginning negotiation
0928 02:10:39.923727 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 02:10:39.923755 (+ 28us) server_negotiation.cc:407] Connection header received
0928 02:10:39.923846 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:10:39.923855 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:10:39.924124 (+ 269us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:10:39.924619 (+ 495us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:10:39.925171 (+ 552us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:10:39.926705 (+ 1534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:10:39.929939 (+ 3234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:10:39.931634 (+ 1695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:10:39.931789 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:10:39.932052 (+ 263us) server_negotiation.cc:1035] Waiting for connection context
0928 02:10:39.933128 (+ 1076us) server_negotiation.cc:299] Negotiation successful
0928 02:10:39.933329 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18875,"thread_start_us":18636,"threads_started":1}
I20260928 02:10:39.934410 8132 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:10:39.904388 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.235.67:35003 (local address 127.0.0.1:58308)
0928 02:10:39.915944 (+ 11556us) negotiation.cc:107] Waiting for socket to connect
0928 02:10:39.916000 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0928 02:10:39.916203 (+ 203us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:10:39.924362 (+ 8159us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:10:39.924366 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:10:39.924382 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:10:39.924962 (+ 580us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:10:39.924980 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:10:39.926862 (+ 1882us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:10:39.926871 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:10:39.929714 (+ 2843us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:10:39.929724 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:10:39.932301 (+ 2577us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:10:39.932308 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:10:39.932642 (+ 334us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:10:39.932801 (+ 159us) client_negotiation.cc:769] Sending connection context
0928 02:10:39.932922 (+ 121us) client_negotiation.cc:240] Negotiation successful
0928 02:10:39.934048 (+ 1126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11258,"mutex_wait_us":1144,"thread_start_us":18584,"threads_started":1}
I20260928 02:10:39.936160 7920 tablet_service.cc:1467] Processing CreateTablet for tablet e8c26902d48342399065156e32d067d3 (DEFAULT_TABLE table=test-table [id=f6ed5b203f7940e8b95989b129f4bd24]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:10:39.937703 7920 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e8c26902d48342399065156e32d067d3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:10:39.938253 8054 tablet_service.cc:1467] Processing CreateTablet for tablet e8c26902d48342399065156e32d067d3 (DEFAULT_TABLE table=test-table [id=f6ed5b203f7940e8b95989b129f4bd24]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:10:39.939688 8054 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e8c26902d48342399065156e32d067d3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:10:39.944169 7786 tablet_service.cc:1467] Processing CreateTablet for tablet e8c26902d48342399065156e32d067d3 (DEFAULT_TABLE table=test-table [id=f6ed5b203f7940e8b95989b129f4bd24]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:10:39.945384 7786 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e8c26902d48342399065156e32d067d3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:10:39.952584 8137 tablet_bootstrap.cc:492] T e8c26902d48342399065156e32d067d3 P 5c0ce3b24633497da7336aa4b455d5b1: Bootstrap starting.
I20260928 02:10:39.954813 8138 tablet_bootstrap.cc:492] T e8c26902d48342399065156e32d067d3 P db0e95f10adf4bb9aab02896f5cc77a0: Bootstrap starting.
I20260928 02:10:39.956463 8137 tablet_bootstrap.cc:654] T e8c26902d48342399065156e32d067d3 P 5c0ce3b24633497da7336aa4b455d5b1: Neither blocks nor log segments found. Creating new log.
I20260928 02:10:39.957795 8139 tablet_bootstrap.cc:492] T e8c26902d48342399065156e32d067d3 P d7bf36f0edb54e33973d0c5f7c82f3ad: Bootstrap starting.
I20260928 02:10:39.958308 8137 log.cc:824] T e8c26902d48342399065156e32d067d3 P 5c0ce3b24633497da7336aa4b455d5b1: Log is configured to *not* fsync() on all Append() calls
I20260928 02:10:39.958078 8138 tablet_bootstrap.cc:654] T e8c26902d48342399065156e32d067d3 P db0e95f10adf4bb9aab02896f5cc77a0: Neither blocks nor log segments found. Creating new log.
I20260928 02:10:39.960076 8138 log.cc:824] T e8c26902d48342399065156e32d067d3 P db0e95f10adf4bb9aab02896f5cc77a0: Log is configured to *not* fsync() on all Append() calls
I20260928 02:10:39.961370 8139 tablet_bootstrap.cc:654] T e8c26902d48342399065156e32d067d3 P d7bf36f0edb54e33973d0c5f7c82f3ad: Neither blocks nor log segments found. Creating new log.
I20260928 02:10:39.962602 8139 log.cc:824] T e8c26902d48342399065156e32d067d3 P d7bf36f0edb54e33973d0c5f7c82f3ad: Log is configured to *not* fsync() on all Append() calls
I20260928 02:10:39.962854 8138 tablet_bootstrap.cc:492] T e8c26902d48342399065156e32d067d3 P db0e95f10adf4bb9aab02896f5cc77a0: No bootstrap required, opened a new log
I20260928 02:10:39.962898 8137 tablet_bootstrap.cc:492] T e8c26902d48342399065156e32d067d3 P 5c0ce3b24633497da7336aa4b455d5b1: No bootstrap required, opened a new log
I20260928 02:10:39.963164 8138 ts_tablet_manager.cc:1397] T e8c26902d48342399065156e32d067d3 P db0e95f10adf4bb9aab02896f5cc77a0: Time spent bootstrapping tablet: real 0.009s user 0.000s sys 0.005s
I20260928 02:10:39.963209 8137 ts_tablet_manager.cc:1397] T e8c26902d48342399065156e32d067d3 P 5c0ce3b24633497da7336aa4b455d5b1: Time spent bootstrapping tablet: real 0.011s user 0.005s sys 0.005s
I20260928 02:10:39.965485 8139 tablet_bootstrap.cc:492] T e8c26902d48342399065156e32d067d3 P d7bf36f0edb54e33973d0c5f7c82f3ad: No bootstrap required, opened a new log
I20260928 02:10:39.965736 8139 ts_tablet_manager.cc:1397] T e8c26902d48342399065156e32d067d3 P d7bf36f0edb54e33973d0c5f7c82f3ad: Time spent bootstrapping tablet: real 0.008s user 0.007s sys 0.000s
I20260928 02:10:39.970664 8138 raft_consensus.cc:348] T e8c26902d48342399065156e32d067d3 P db0e95f10adf4bb9aab02896f5cc77a0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db0e95f10adf4bb9aab02896f5cc77a0" member_type: VOTER last_known_addr { host: "127.6.235.67" port: 35003 } } peers { permanent_uuid: "d7bf36f0edb54e33973d0c5f7c82f3ad" member_type: VOTER last_known_addr { host: "127.6.235.65" port: 42579 } } peers { permanent_uuid: "5c0ce3b24633497da7336aa4b455d5b1" member_type: VOTER last_known_addr { host: "127.6.235.66" port: 42755 } }
I20260928 02:10:39.971117 8138 raft_consensus.cc:374] T e8c26902d48342399065156e32d067d3 P db0e95f10adf4bb9aab02896f5cc77a0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:10:39.971241 8138 raft_consensus.cc:729] T e8c26902d48342399065156e32d067d3 P db0e95f10adf4bb9aab02896f5cc77a0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db0e95f10adf4bb9aab02896f5cc77a0, State: Initialized, Role: FOLLOWER
I20260928 02:10:39.971882 8138 consensus_queue.cc:260] T e8c26902d48342399065156e32d067d3 P db0e95f10adf4bb9aab02896f5cc77a0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db0e95f10adf4bb9aab02896f5cc77a0" member_type: VOTER last_known_addr { host: "127.6.235.67" port: 35003 } } peers { permanent_uuid: "d7bf36f0edb54e33973d0c5f7c82f3ad" member_type: VOTER last_known_addr { host: "127.6.235.65" port: 42579 } } peers { permanent_uuid: "5c0ce3b24633497da7336aa4b455d5b1" member_type: VOTER last_known_addr { host: "127.6.235.66" port: 42755 } }
I20260928 02:10:39.973067 8119 heartbeater.cc:503] Master 127.6.235.126:43611 was elected leader, sending a full tablet report...
I20260928 02:10:39.973655 8138 ts_tablet_manager.cc:1428] T e8c26902d48342399065156e32d067d3 P db0e95f10adf4bb9aab02896f5cc77a0: Time spent starting tablet: real 0.010s user 0.011s sys 0.001s
I20260928 02:10:39.974128 8137 raft_consensus.cc:348] T e8c26902d48342399065156e32d067d3 P 5c0ce3b24633497da7336aa4b455d5b1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db0e95f10adf4bb9aab02896f5cc77a0" member_type: VOTER last_known_addr { host: "127.6.235.67" port: 35003 } } peers { permanent_uuid: "d7bf36f0edb54e33973d0c5f7c82f3ad" member_type: VOTER last_known_addr { host: "127.6.235.65" port: 42579 } } peers { permanent_uuid: "5c0ce3b24633497da7336aa4b455d5b1" member_type: VOTER last_known_addr { host: "127.6.235.66" port: 42755 } }
I20260928 02:10:39.974571 8137 raft_consensus.cc:374] T e8c26902d48342399065156e32d067d3 P 5c0ce3b24633497da7336aa4b455d5b1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:10:39.974692 8137 raft_consensus.cc:729] T e8c26902d48342399065156e32d067d3 P 5c0ce3b24633497da7336aa4b455d5b1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5c0ce3b24633497da7336aa4b455d5b1, State: Initialized, Role: FOLLOWER
I20260928 02:10:39.975342 8137 consensus_queue.cc:260] T e8c26902d48342399065156e32d067d3 P 5c0ce3b24633497da7336aa4b455d5b1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db0e95f10adf4bb9aab02896f5cc77a0" member_type: VOTER last_known_addr { host: "127.6.235.67" port: 35003 } } peers { permanent_uuid: "d7bf36f0edb54e33973d0c5f7c82f3ad" member_type: VOTER last_known_addr { host: "127.6.235.65" port: 42579 } } peers { permanent_uuid: "5c0ce3b24633497da7336aa4b455d5b1" member_type: VOTER last_known_addr { host: "127.6.235.66" port: 42755 } }
I20260928 02:10:39.976425 7985 heartbeater.cc:503] Master 127.6.235.126:43611 was elected leader, sending a full tablet report...
I20260928 02:10:39.976732 8139 raft_consensus.cc:348] T e8c26902d48342399065156e32d067d3 P d7bf36f0edb54e33973d0c5f7c82f3ad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db0e95f10adf4bb9aab02896f5cc77a0" member_type: VOTER last_known_addr { host: "127.6.235.67" port: 35003 } } peers { permanent_uuid: "d7bf36f0edb54e33973d0c5f7c82f3ad" member_type: VOTER last_known_addr { host: "127.6.235.65" port: 42579 } } peers { permanent_uuid: "5c0ce3b24633497da7336aa4b455d5b1" member_type: VOTER last_known_addr { host: "127.6.235.66" port: 42755 } }
I20260928 02:10:39.977174 8137 ts_tablet_manager.cc:1428] T e8c26902d48342399065156e32d067d3 P 5c0ce3b24633497da7336aa4b455d5b1: Time spent starting tablet: real 0.014s user 0.014s sys 0.000s
I20260928 02:10:39.977180 8139 raft_consensus.cc:374] T e8c26902d48342399065156e32d067d3 P d7bf36f0edb54e33973d0c5f7c82f3ad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:10:39.977272 8139 raft_consensus.cc:729] T e8c26902d48342399065156e32d067d3 P d7bf36f0edb54e33973d0c5f7c82f3ad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d7bf36f0edb54e33973d0c5f7c82f3ad, State: Initialized, Role: FOLLOWER
I20260928 02:10:39.977910 8139 consensus_queue.cc:260] T e8c26902d48342399065156e32d067d3 P d7bf36f0edb54e33973d0c5f7c82f3ad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db0e95f10adf4bb9aab02896f5cc77a0" member_type: VOTER last_known_addr { host: "127.6.235.67" port: 35003 } } peers { permanent_uuid: "d7bf36f0edb54e33973d0c5f7c82f3ad" member_type: VOTER last_known_addr { host: "127.6.235.65" port: 42579 } } peers { permanent_uuid: "5c0ce3b24633497da7336aa4b455d5b1" member_type: VOTER last_known_addr { host: "127.6.235.66" port: 42755 } }
I20260928 02:10:39.979512 8139 ts_tablet_manager.cc:1428] T e8c26902d48342399065156e32d067d3 P d7bf36f0edb54e33973d0c5f7c82f3ad: Time spent starting tablet: real 0.014s user 0.013s sys 0.000s
W20260928 02:10:40.024781 7852 tablet.cc:2369] T e8c26902d48342399065156e32d067d3 P d7bf36f0edb54e33973d0c5f7c82f3ad: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 02:10:40.049583 8144 raft_consensus.cc:482] T e8c26902d48342399065156e32d067d3 P 5c0ce3b24633497da7336aa4b455d5b1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 02:10:40.049898 8144 raft_consensus.cc:504] T e8c26902d48342399065156e32d067d3 P 5c0ce3b24633497da7336aa4b455d5b1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db0e95f10adf4bb9aab02896f5cc77a0" member_type: VOTER last_known_addr { host: "127.6.235.67" port: 35003 } } peers { permanent_uuid: "d7bf36f0edb54e33973d0c5f7c82f3ad" member_type: VOTER last_known_addr { host: "127.6.235.65" port: 42579 } } peers { permanent_uuid: "5c0ce3b24633497da7336aa4b455d5b1" member_type: VOTER last_known_addr { host: "127.6.235.66" port: 42755 } }
I20260928 02:10:40.051681 8144 leader_election.cc:288] T e8c26902d48342399065156e32d067d3 P 5c0ce3b24633497da7336aa4b455d5b1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers db0e95f10adf4bb9aab02896f5cc77a0 (127.6.235.67:35003), d7bf36f0edb54e33973d0c5f7c82f3ad (127.6.235.65:42579)
W20260928 02:10:40.056706 7986 tablet.cc:2369] T e8c26902d48342399065156e32d067d3 P 5c0ce3b24633497da7336aa4b455d5b1: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 02:10:40.059254 8133 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:10:40.051958 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.235.66:39575 (local address 127.6.235.67:35003)
0928 02:10:40.052173 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0928 02:10:40.052180 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 02:10:40.052969 (+ 789us) server_negotiation.cc:407] Connection header received
0928 02:10:40.053262 (+ 293us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:10:40.053269 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:10:40.053344 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:10:40.053410 (+ 66us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:10:40.054078 (+ 668us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:10:40.054825 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:10:40.056836 (+ 2011us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:10:40.058157 (+ 1321us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:10:40.058298 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:10:40.058353 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0928 02:10:40.058883 (+ 530us) server_negotiation.cc:299] Negotiation successful
0928 02:10:40.059033 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260928 02:10:40.059242 8134 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:10:40.052252 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.235.66:44517 (local address 127.6.235.65:42579)
0928 02:10:40.052471 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0928 02:10:40.052480 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 02:10:40.053141 (+ 661us) server_negotiation.cc:407] Connection header received
0928 02:10:40.053369 (+ 228us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:10:40.053374 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:10:40.053464 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:10:40.053577 (+ 113us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:10:40.054162 (+ 585us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:10:40.054928 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:10:40.056697 (+ 1769us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:10:40.058003 (+ 1306us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:10:40.058174 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:10:40.058263 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0928 02:10:40.058855 (+ 592us) server_negotiation.cc:299] Negotiation successful
0928 02:10:40.058988 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260928 02:10:40.059628 8147 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:10:40.052121 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.235.65:42579 (local address 127.6.235.66:44517)
0928 02:10:40.053020 (+ 899us) negotiation.cc:107] Waiting for socket to connect
0928 02:10:40.053034 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 02:10:40.053189 (+ 155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:10:40.053614 (+ 425us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:10:40.053618 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:10:40.053657 (+ 39us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:10:40.053992 (+ 335us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:10:40.054026 (+ 34us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:10:40.055066 (+ 1040us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:10:40.055070 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:10:40.056543 (+ 1473us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:10:40.056554 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:10:40.058282 (+ 1728us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:10:40.058290 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:10:40.058611 (+ 321us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:10:40.058683 (+ 72us) client_negotiation.cc:769] Sending connection context
0928 02:10:40.058813 (+ 130us) client_negotiation.cc:240] Negotiation successful
0928 02:10:40.059027 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":798,"mutex_wait_us":95,"thread_start_us":563,"threads_started":1}
I20260928 02:10:40.059652 8146 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:10:40.051933 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.235.67:35003 (local address 127.6.235.66:39575)
0928 02:10:40.052831 (+ 898us) negotiation.cc:107] Waiting for socket to connect
0928 02:10:40.052851 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0928 02:10:40.053048 (+ 197us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:10:40.053452 (+ 404us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:10:40.053458 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:10:40.053483 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:10:40.053937 (+ 454us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:10:40.053947 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:10:40.054947 (+ 1000us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:10:40.054953 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:10:40.056612 (+ 1659us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:10:40.056629 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:10:40.058470 (+ 1841us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:10:40.058478 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:10:40.058700 (+ 222us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:10:40.058742 (+ 42us) client_negotiation.cc:769] Sending connection context
0928 02:10:40.058846 (+ 104us) client_negotiation.cc:240] Negotiation successful
0928 02:10:40.059025 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":738,"mutex_wait_us":64,"thread_start_us":484,"threads_started":1}
I20260928 02:10:40.060806 7806 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e8c26902d48342399065156e32d067d3" candidate_uuid: "5c0ce3b24633497da7336aa4b455d5b1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d7bf36f0edb54e33973d0c5f7c82f3ad" is_pre_election: true
I20260928 02:10:40.060768 8074 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e8c26902d48342399065156e32d067d3" candidate_uuid: "5c0ce3b24633497da7336aa4b455d5b1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db0e95f10adf4bb9aab02896f5cc77a0" is_pre_election: true
I20260928 02:10:40.061404 7806 raft_consensus.cc:2454] T e8c26902d48342399065156e32d067d3 P d7bf36f0edb54e33973d0c5f7c82f3ad [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5c0ce3b24633497da7336aa4b455d5b1 in term 0.
I20260928 02:10:40.061434 8074 raft_consensus.cc:2454] T e8c26902d48342399065156e32d067d3 P db0e95f10adf4bb9aab02896f5cc77a0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5c0ce3b24633497da7336aa4b455d5b1 in term 0.
I20260928 02:10:40.062090 7874 leader_election.cc:302] T e8c26902d48342399065156e32d067d3 P 5c0ce3b24633497da7336aa4b455d5b1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5c0ce3b24633497da7336aa4b455d5b1, d7bf36f0edb54e33973d0c5f7c82f3ad; no voters:
I20260928 02:10:40.062480 8144 raft_consensus.cc:2781] T e8c26902d48342399065156e32d067d3 P 5c0ce3b24633497da7336aa4b455d5b1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 02:10:40.062634 8144 raft_consensus.cc:482] T e8c26902d48342399065156e32d067d3 P 5c0ce3b24633497da7336aa4b455d5b1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 02:10:40.062733 8144 raft_consensus.cc:3037] T e8c26902d48342399065156e32d067d3 P 5c0ce3b24633497da7336aa4b455d5b1 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:10:40.064805 8144 raft_consensus.cc:504] T e8c26902d48342399065156e32d067d3 P 5c0ce3b24633497da7336aa4b455d5b1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db0e95f10adf4bb9aab02896f5cc77a0" member_type: VOTER last_known_addr { host: "127.6.235.67" port: 35003 } } peers { permanent_uuid: "d7bf36f0edb54e33973d0c5f7c82f3ad" member_type: VOTER last_known_addr { host: "127.6.235.65" port: 42579 } } peers { permanent_uuid: "5c0ce3b24633497da7336aa4b455d5b1" member_type: VOTER last_known_addr { host: "127.6.235.66" port: 42755 } }
I20260928 02:10:40.065439 8144 leader_election.cc:288] T e8c26902d48342399065156e32d067d3 P 5c0ce3b24633497da7336aa4b455d5b1 [CANDIDATE]: Term 1 election: Requested vote from peers db0e95f10adf4bb9aab02896f5cc77a0 (127.6.235.67:35003), d7bf36f0edb54e33973d0c5f7c82f3ad (127.6.235.65:42579)
I20260928 02:10:40.065891 7806 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e8c26902d48342399065156e32d067d3" candidate_uuid: "5c0ce3b24633497da7336aa4b455d5b1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d7bf36f0edb54e33973d0c5f7c82f3ad"
I20260928 02:10:40.065892 8074 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e8c26902d48342399065156e32d067d3" candidate_uuid: "5c0ce3b24633497da7336aa4b455d5b1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db0e95f10adf4bb9aab02896f5cc77a0"
I20260928 02:10:40.066095 7806 raft_consensus.cc:3037] T e8c26902d48342399065156e32d067d3 P d7bf36f0edb54e33973d0c5f7c82f3ad [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:10:40.066095 8074 raft_consensus.cc:3037] T e8c26902d48342399065156e32d067d3 P db0e95f10adf4bb9aab02896f5cc77a0 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:10:40.069316 7806 raft_consensus.cc:2454] T e8c26902d48342399065156e32d067d3 P d7bf36f0edb54e33973d0c5f7c82f3ad [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5c0ce3b24633497da7336aa4b455d5b1 in term 1.
I20260928 02:10:40.069316 8074 raft_consensus.cc:2454] T e8c26902d48342399065156e32d067d3 P db0e95f10adf4bb9aab02896f5cc77a0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5c0ce3b24633497da7336aa4b455d5b1 in term 1.
I20260928 02:10:40.069741 7874 leader_election.cc:302] T e8c26902d48342399065156e32d067d3 P 5c0ce3b24633497da7336aa4b455d5b1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5c0ce3b24633497da7336aa4b455d5b1, d7bf36f0edb54e33973d0c5f7c82f3ad; no voters:
I20260928 02:10:40.070000 8144 raft_consensus.cc:2781] T e8c26902d48342399065156e32d067d3 P 5c0ce3b24633497da7336aa4b455d5b1 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:10:40.070432 8144 raft_consensus.cc:686] T e8c26902d48342399065156e32d067d3 P 5c0ce3b24633497da7336aa4b455d5b1 [term 1 LEADER]: Becoming Leader. State: Replica: 5c0ce3b24633497da7336aa4b455d5b1, State: Running, Role: LEADER
I20260928 02:10:40.070923 8144 consensus_queue.cc:237] T e8c26902d48342399065156e32d067d3 P 5c0ce3b24633497da7336aa4b455d5b1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db0e95f10adf4bb9aab02896f5cc77a0" member_type: VOTER last_known_addr { host: "127.6.235.67" port: 35003 } } peers { permanent_uuid: "d7bf36f0edb54e33973d0c5f7c82f3ad" member_type: VOTER last_known_addr { host: "127.6.235.65" port: 42579 } } peers { permanent_uuid: "5c0ce3b24633497da7336aa4b455d5b1" member_type: VOTER last_known_addr { host: "127.6.235.66" port: 42755 } }
W20260928 02:10:40.072700 8120 tablet.cc:2369] T e8c26902d48342399065156e32d067d3 P db0e95f10adf4bb9aab02896f5cc77a0: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 02:10:40.075614 7662 catalog_manager.cc:5495] T e8c26902d48342399065156e32d067d3 P 5c0ce3b24633497da7336aa4b455d5b1 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5c0ce3b24633497da7336aa4b455d5b1 (127.6.235.66). New cstate: current_term: 1 leader_uuid: "5c0ce3b24633497da7336aa4b455d5b1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "db0e95f10adf4bb9aab02896f5cc77a0" member_type: VOTER last_known_addr { host: "127.6.235.67" port: 35003 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d7bf36f0edb54e33973d0c5f7c82f3ad" member_type: VOTER last_known_addr { host: "127.6.235.65" port: 42579 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5c0ce3b24633497da7336aa4b455d5b1" member_type: VOTER last_known_addr { host: "127.6.235.66" port: 42755 } health_report { overall_health: HEALTHY } } }
Sep 28 02:10:40 dist-test-slave-86gv krb5kdc[7617](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561437, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.235.65@KRBTEST.COM
I20260928 02:10:40.127597 8134 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:10:40.112747 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48100 (local address 127.6.235.65:42579)
0928 02:10:40.113012 (+ 265us) server_negotiation.cc:206] Beginning negotiation
0928 02:10:40.113022 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 02:10:40.113442 (+ 420us) server_negotiation.cc:407] Connection header received
0928 02:10:40.113576 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:10:40.113579 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:10:40.113642 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:10:40.113724 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:10:40.115173 (+ 1449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:10:40.116223 (+ 1050us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:10:40.117585 (+ 1362us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:10:40.118080 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:10:40.118197 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:10:40.122162 (+ 3965us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:10:40.122176 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:10:40.122193 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:10:40.122232 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:10:40.124860 (+ 2628us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:10:40.125509 (+ 649us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:10:40.125541 (+ 32us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:10:40.125543 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:10:40.125666 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:10:40.126010 (+ 344us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:10:40.126015 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:10:40.126023 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:10:40.126444 (+ 421us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:10:40.126716 (+ 272us) server_negotiation.cc:1035] Waiting for connection context
0928 02:10:40.126893 (+ 177us) server_negotiation.cc:299] Negotiation successful
0928 02:10:40.127073 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
Sep 28 02:10:40 dist-test-slave-86gv krb5kdc[7617](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561440, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 28 02:10:40 dist-test-slave-86gv krb5kdc[7617](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561440, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.6.235.65@KRBTEST.COM
I20260928 02:10:40.174441 8134 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:10:40.160713 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48114 (local address 127.6.235.65:42579)
0928 02:10:40.160955 (+ 242us) server_negotiation.cc:206] Beginning negotiation
0928 02:10:40.160964 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 02:10:40.161295 (+ 331us) server_negotiation.cc:407] Connection header received
0928 02:10:40.161504 (+ 209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:10:40.161514 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:10:40.161603 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:10:40.161710 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:10:40.163048 (+ 1338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:10:40.163793 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:10:40.165099 (+ 1306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:10:40.165504 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:10:40.165645 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:10:40.169355 (+ 3710us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:10:40.169360 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:10:40.169363 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:10:40.169397 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:10:40.171829 (+ 2432us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:10:40.172465 (+ 636us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:10:40.172470 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:10:40.172473 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:10:40.172602 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:10:40.173092 (+ 490us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:10:40.173101 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:10:40.173103 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:10:40.173360 (+ 257us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:10:40.173655 (+ 295us) server_negotiation.cc:1035] Waiting for connection context
0928 02:10:40.173906 (+ 251us) server_negotiation.cc:299] Negotiation successful
0928 02:10:40.174116 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
Sep 28 02:10:40 dist-test-slave-86gv krb5kdc[7617](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561440, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 28 02:10:40 dist-test-slave-86gv krb5kdc[7617](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561440, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.235.65@KRBTEST.COM
I20260928 02:10:40.220549 8134 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:10:40.206321 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48128 (local address 127.6.235.65:42579)
0928 02:10:40.206576 (+ 255us) server_negotiation.cc:206] Beginning negotiation
0928 02:10:40.206584 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 02:10:40.206876 (+ 292us) server_negotiation.cc:407] Connection header received
0928 02:10:40.207099 (+ 223us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:10:40.207103 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:10:40.207165 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:10:40.207267 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:10:40.208554 (+ 1287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:10:40.209315 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:10:40.210597 (+ 1282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:10:40.211008 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:10:40.211174 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:10:40.215094 (+ 3920us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:10:40.215099 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:10:40.215102 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:10:40.215134 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:10:40.217937 (+ 2803us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:10:40.218578 (+ 641us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:10:40.218584 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:10:40.218587 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:10:40.218709 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:10:40.219099 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:10:40.219106 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:10:40.219109 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:10:40.219409 (+ 300us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:10:40.219672 (+ 263us) server_negotiation.cc:1035] Waiting for connection context
0928 02:10:40.219967 (+ 295us) server_negotiation.cc:299] Negotiation successful
0928 02:10:40.220195 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
W20260928 02:10:40.222312 7766 tablet_service.cc:2908] Rejecting scan request for tablet e8c26902d48342399065156e32d067d3: Uninitialized: clean time has not yet been initialized
I20260928 02:10:40.225634 7085 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskTlaUuc/build/asan/bin/kudu pid 7723
I20260928 02:10:40.238515 8134 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:10:40.226436 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48140 (local address 127.6.235.65:42579)
0928 02:10:40.226595 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0928 02:10:40.226599 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:10:40.226995 (+ 396us) server_negotiation.cc:407] Connection header received
0928 02:10:40.227104 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:10:40.227108 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:10:40.227209 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:10:40.227313 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:10:40.228448 (+ 1135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:10:40.229358 (+ 910us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:10:40.230487 (+ 1129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:10:40.230882 (+ 395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:10:40.230984 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:10:40.233232 (+ 2248us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:10:40.233239 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:10:40.233244 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:10:40.233305 (+ 61us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:10:40.235965 (+ 2660us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:10:40.236618 (+ 653us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:10:40.236622 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:10:40.236624 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:10:40.236741 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:10:40.237209 (+ 468us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:10:40.237213 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:10:40.237217 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:10:40.237555 (+ 338us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:10:40.237770 (+ 215us) server_negotiation.cc:1035] Waiting for connection context
0928 02:10:40.238023 (+ 253us) server_negotiation.cc:299] Negotiation successful
0928 02:10:40.238187 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260928 02:10:40.239203 7846 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 02:10:40.330163 7085 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTlaUuc/build/asan/bin/kudu with pid 7723
I20260928 02:10:40.348009 7085 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskTlaUuc/build/asan/bin/kudu pid 7857
Sep 28 02:10:40 dist-test-slave-86gv krb5kdc[7617](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561440, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.235.66@KRBTEST.COM
I20260928 02:10:40.362928 8136 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:10:40.348883 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45854 (local address 127.6.235.66:42755)
0928 02:10:40.349132 (+ 249us) server_negotiation.cc:206] Beginning negotiation
0928 02:10:40.349139 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 02:10:40.349158 (+ 19us) server_negotiation.cc:407] Connection header received
0928 02:10:40.349257 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:10:40.349262 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:10:40.349320 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:10:40.349430 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:10:40.350563 (+ 1133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:10:40.351269 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:10:40.352526 (+ 1257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:10:40.353118 (+ 592us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:10:40.353251 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:10:40.357408 (+ 4157us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:10:40.357423 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:10:40.357440 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:10:40.357482 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:10:40.360110 (+ 2628us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:10:40.360796 (+ 686us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:10:40.360812 (+ 16us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:10:40.360814 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:10:40.361037 (+ 223us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:10:40.361396 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:10:40.361402 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:10:40.361405 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:10:40.361768 (+ 363us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:10:40.362073 (+ 305us) server_negotiation.cc:1035] Waiting for connection context
0928 02:10:40.362350 (+ 277us) server_negotiation.cc:299] Negotiation successful
0928 02:10:40.362577 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260928 02:10:40.363665 7980 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 02:10:40.454908 7085 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTlaUuc/build/asan/bin/kudu with pid 7857
I20260928 02:10:40.472689 7085 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskTlaUuc/build/asan/bin/kudu pid 7991
Sep 28 02:10:40 dist-test-slave-86gv krb5kdc[7617](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561440, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.235.67@KRBTEST.COM
I20260928 02:10:40.488042 8133 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:10:40.473538 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58324 (local address 127.6.235.67:35003)
0928 02:10:40.473728 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0928 02:10:40.473733 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 02:10:40.473765 (+ 32us) server_negotiation.cc:407] Connection header received
0928 02:10:40.473915 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:10:40.473919 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:10:40.473989 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:10:40.474082 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:10:40.475423 (+ 1341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:10:40.476130 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:10:40.477692 (+ 1562us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:10:40.478125 (+ 433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:10:40.478255 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:10:40.482450 (+ 4195us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:10:40.482463 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:10:40.482479 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:10:40.482514 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:10:40.485166 (+ 2652us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:10:40.486152 (+ 986us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:10:40.486159 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:10:40.486161 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:10:40.486298 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:10:40.486723 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:10:40.486726 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:10:40.486728 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:10:40.487069 (+ 341us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:10:40.487267 (+ 198us) server_negotiation.cc:1035] Waiting for connection context
0928 02:10:40.487516 (+ 249us) server_negotiation.cc:299] Negotiation successful
0928 02:10:40.487670 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260928 02:10:40.488835 8114 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 02:10:40.579660 7085 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTlaUuc/build/asan/bin/kudu with pid 7991
I20260928 02:10:40.596534 7085 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskTlaUuc/build/asan/bin/kudu pid 7630
I20260928 02:10:40.597673 7692 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 02:10:40.709323 7085 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTlaUuc/build/asan/bin/kudu with pid 7630
2026-09-28T02:10:40Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnChecksum (4268 ms)
[ RUN ] SecurityITest.SmokeTestAsAuthorizedUser
Loading random data
Initializing database '/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790561432251112-7085-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 02:10:40 dist-test-slave-86gv krb5kdc[8167](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 02:10:40 dist-test-slave-86gv krb5kdc[8167](info): set up 2 sockets
Sep 28 02:10:40 dist-test-slave-86gv krb5kdc[8167](info): commencing operation
krb5kdc: starting...
W20260928 02:10:41.853492 7085 mini_kdc.cc:121] Time spent starting KDC: real 1.100s user 0.004s sys 0.013s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790561432251112-7085-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790561432251112-7085-0/krb5kdc/test-user.keytab.
Sep 28 02:10:41 dist-test-slave-86gv krb5kdc[8167](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561441, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T02:10:41Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T02:10:41Z Disabled control of system clock
WARNING: no policy specified for kudu/127.6.235.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.235.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.235.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790561432251112-7085-0/krb5kdc/kudu_127.6.235.126.keytab.
Entry for principal kudu/127.6.235.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790561432251112-7085-0/krb5kdc/kudu_127.6.235.126.keytab.
I20260928 02:10:42.052881 7085 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTlaUuc/build/asan/bin/kudu
/tmp/dist-test-taskTlaUuc/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790561432251112-7085-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790561432251112-7085-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790561432251112-7085-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790561432251112-7085-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.235.126:37431
--webserver_interface=127.6.235.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790561432251112-7085-0/krb5kdc/kudu_127.6.235.126.keytab
--principal=kudu/127.6.235.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.235.84:34167
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.235.126:37431
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790561432251112-7085-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790561432251112-7085-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790561432251112-7085-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:10:42.386050 8180 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:10:42.386462 8180 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:10:42.386620 8180 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:10:42.394132 8180 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 02:10:42.394282 8180 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260928 02:10:42.394362 8180 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:10:42.394452 8180 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:10:42.394542 8180 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 02:10:42.394629 8180 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 02:10:42.404675 8180 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.235.84:34167
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790561432251112-7085-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790561432251112-7085-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.235.126:37431
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790561432251112-7085-0/krb5kdc/kudu_127.6.235.126.keytab
--principal=kudu/127.6.235.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.235.126:37431
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790561432251112-7085-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.235.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790561432251112-7085-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 00:51:59 UTC on ccycloud.quasar-axgkwn.root.comops.site
build id 4676464
ASAN enabled
I20260928 02:10:42.406111 8180 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:10:42.407543 8180 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:10:42.419812 8185 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:10:42.419727 8188 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:10:42.419847 8186 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:10:42.420437 8180 server_base.cc:1034] running on GCE node
I20260928 02:10:42.421298 8180 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:10:42.422674 8180 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:10:42.423910 8180 hybrid_clock.cc:648] HybridClock initialized: now 1790561442423852 us; error 30 us; skew 500 ppm
I20260928 02:10:42.424219 8180 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 02:10:42 dist-test-slave-86gv krb5kdc[8167](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561442, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.235.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:10:42.427536 8180 init.cc:377] Logged in from keytab as kudu/127.6.235.126@KRBTEST.COM (short username kudu)
I20260928 02:10:42.429106 8180 webserver.cc:460] Webserver started at http://127.6.235.126:35027/ using document root <none> and password file <none>
I20260928 02:10:42.429827 8180 fs_manager.cc:359] Metadata directory not provided
I20260928 02:10:42.429922 8180 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:10:42.430212 8180 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:10:42.433350 8180 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790561432251112-7085-0/minicluster-data/master-0/data/instance:
uuid: "ae6ca031c9744a50b87bd00e3e3df444"
format_stamp: "Formatted at 2026-09-28 02:10:42 on dist-test-slave-86gv"
server_key: "50ddf9f4531ec52428418bc8877a6255"
server_key_iv: "eb9925d74385563874aaa5ef1cfde4cd"
server_key_version: "encryptionkey@0"
I20260928 02:10:42.434190 8180 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790561432251112-7085-0/minicluster-data/master-0/wal/instance:
uuid: "ae6ca031c9744a50b87bd00e3e3df444"
format_stamp: "Formatted at 2026-09-28 02:10:42 on dist-test-slave-86gv"
server_key: "50ddf9f4531ec52428418bc8877a6255"
server_key_iv: "eb9925d74385563874aaa5ef1cfde4cd"
server_key_version: "encryptionkey@0"
I20260928 02:10:42.439775 8180 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260928 02:10:42.443466 8195 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:10:42.444878 8180 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260928 02:10:42.445152 8180 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790561432251112-7085-0/minicluster-data/master-0/data,/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790561432251112-7085-0/minicluster-data/master-0/wal
uuid: "ae6ca031c9744a50b87bd00e3e3df444"
format_stamp: "Formatted at 2026-09-28 02:10:42 on dist-test-slave-86gv"
server_key: "50ddf9f4531ec52428418bc8877a6255"
server_key_iv: "eb9925d74385563874aaa5ef1cfde4cd"
server_key_version: "encryptionkey@0"
I20260928 02:10:42.445385 8180 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790561432251112-7085-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790561432251112-7085-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790561432251112-7085-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:10:42.468295 8180 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:10:42.472795 8180 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:10:42.473130 8180 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:10:42.494822 8180 rpc_server.cc:307] RPC server started. Bound to: 127.6.235.126:37431
I20260928 02:10:42.494829 8256 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.235.126:37431 every 8 connection(s)
I20260928 02:10:42.496286 8180 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790561432251112-7085-0/minicluster-data/master-0/data/info.pb
I20260928 02:10:42.500339 8257 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:10:42.501381 7085 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTlaUuc/build/asan/bin/kudu as pid 8180
I20260928 02:10:42.501603 7085 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790561432251112-7085-0/minicluster-data/master-0/wal/instance
I20260928 02:10:42.502482 7085 external_mini_cluster.cc:1428] Setting key 7af7d3de7934ef0e026ba1e2ad50487f
I20260928 02:10:42.515275 8257 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ae6ca031c9744a50b87bd00e3e3df444: Bootstrap starting.
Sep 28 02:10:42 dist-test-slave-86gv krb5kdc[8167](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561441, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.235.126@KRBTEST.COM
I20260928 02:10:42.518972 8257 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ae6ca031c9744a50b87bd00e3e3df444: Neither blocks nor log segments found. Creating new log.
I20260928 02:10:42.520051 8257 log.cc:824] T 00000000000000000000000000000000 P ae6ca031c9744a50b87bd00e3e3df444: Log is configured to *not* fsync() on all Append() calls
I20260928 02:10:42.523139 8257 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ae6ca031c9744a50b87bd00e3e3df444: No bootstrap required, opened a new log
I20260928 02:10:42.525988 8260 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:10:42.504554 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46598 (local address 127.6.235.126:37431)
0928 02:10:42.505375 (+ 821us) server_negotiation.cc:206] Beginning negotiation
0928 02:10:42.505386 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 02:10:42.505432 (+ 46us) server_negotiation.cc:407] Connection header received
0928 02:10:42.507040 (+ 1608us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:10:42.507067 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:10:42.507510 (+ 443us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:10:42.507875 (+ 365us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:10:42.509684 (+ 1809us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:10:42.510990 (+ 1306us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:10:42.512780 (+ 1790us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:10:42.513630 (+ 850us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:10:42.513762 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:10:42.518369 (+ 4607us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:10:42.518377 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:10:42.518393 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:10:42.518432 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:10:42.521992 (+ 3560us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:10:42.522685 (+ 693us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:10:42.522692 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:10:42.522698 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:10:42.522860 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:10:42.523242 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:10:42.523250 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:10:42.523254 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:10:42.523789 (+ 535us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:10:42.524081 (+ 292us) server_negotiation.cc:1035] Waiting for connection context
0928 02:10:42.524505 (+ 424us) server_negotiation.cc:299] Negotiation successful
0928 02:10:42.525002 (+ 497us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":594,"thread_start_us":381,"threads_started":1}
I20260928 02:10:42.531103 8257 raft_consensus.cc:348] T 00000000000000000000000000000000 P ae6ca031c9744a50b87bd00e3e3df444 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ae6ca031c9744a50b87bd00e3e3df444" member_type: VOTER last_known_addr { host: "127.6.235.126" port: 37431 } }
I20260928 02:10:42.531570 8257 raft_consensus.cc:374] T 00000000000000000000000000000000 P ae6ca031c9744a50b87bd00e3e3df444 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:10:42.531700 8257 raft_consensus.cc:729] T 00000000000000000000000000000000 P ae6ca031c9744a50b87bd00e3e3df444 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ae6ca031c9744a50b87bd00e3e3df444, State: Initialized, Role: FOLLOWER
I20260928 02:10:42.532282 8257 consensus_queue.cc:260] T 00000000000000000000000000000000 P ae6ca031c9744a50b87bd00e3e3df444 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ae6ca031c9744a50b87bd00e3e3df444" member_type: VOTER last_known_addr { host: "127.6.235.126" port: 37431 } }
I20260928 02:10:42.532496 8257 raft_consensus.cc:388] T 00000000000000000000000000000000 P ae6ca031c9744a50b87bd00e3e3df444 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 02:10:42.532598 8257 raft_consensus.cc:482] T 00000000000000000000000000000000 P ae6ca031c9744a50b87bd00e3e3df444 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 02:10:42.532717 8257 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ae6ca031c9744a50b87bd00e3e3df444 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:10:42.534861 8257 raft_consensus.cc:504] T 00000000000000000000000000000000 P ae6ca031c9744a50b87bd00e3e3df444 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ae6ca031c9744a50b87bd00e3e3df444" member_type: VOTER last_known_addr { host: "127.6.235.126" port: 37431 } }
I20260928 02:10:42.535324 8257 leader_election.cc:302] T 00000000000000000000000000000000 P ae6ca031c9744a50b87bd00e3e3df444 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ae6ca031c9744a50b87bd00e3e3df444; no voters:
I20260928 02:10:42.535873 8257 leader_election.cc:288] T 00000000000000000000000000000000 P ae6ca031c9744a50b87bd00e3e3df444 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 02:10:42.536038 8262 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ae6ca031c9744a50b87bd00e3e3df444 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:10:42.537092 8262 raft_consensus.cc:686] T 00000000000000000000000000000000 P ae6ca031c9744a50b87bd00e3e3df444 [term 1 LEADER]: Becoming Leader. State: Replica: ae6ca031c9744a50b87bd00e3e3df444, State: Running, Role: LEADER
I20260928 02:10:42.537701 8262 consensus_queue.cc:237] T 00000000000000000000000000000000 P ae6ca031c9744a50b87bd00e3e3df444 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ae6ca031c9744a50b87bd00e3e3df444" member_type: VOTER last_known_addr { host: "127.6.235.126" port: 37431 } }
I20260928 02:10:42.538154 8257 sys_catalog.cc:564] T 00000000000000000000000000000000 P ae6ca031c9744a50b87bd00e3e3df444 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 02:10:42.540753 8264 sys_catalog.cc:455] T 00000000000000000000000000000000 P ae6ca031c9744a50b87bd00e3e3df444 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ae6ca031c9744a50b87bd00e3e3df444. Latest consensus state: current_term: 1 leader_uuid: "ae6ca031c9744a50b87bd00e3e3df444" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ae6ca031c9744a50b87bd00e3e3df444" member_type: VOTER last_known_addr { host: "127.6.235.126" port: 37431 } } }
I20260928 02:10:42.540699 8263 sys_catalog.cc:455] T 00000000000000000000000000000000 P ae6ca031c9744a50b87bd00e3e3df444 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ae6ca031c9744a50b87bd00e3e3df444" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ae6ca031c9744a50b87bd00e3e3df444" member_type: VOTER last_known_addr { host: "127.6.235.126" port: 37431 } } }
I20260928 02:10:42.541157 8264 sys_catalog.cc:458] T 00000000000000000000000000000000 P ae6ca031c9744a50b87bd00e3e3df444 [sys.catalog]: This master's current role is: LEADER
I20260928 02:10:42.541225 8263 sys_catalog.cc:458] T 00000000000000000000000000000000 P ae6ca031c9744a50b87bd00e3e3df444 [sys.catalog]: This master's current role is: LEADER
I20260928 02:10:42.542332 8271 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 02:10:42.547719 8271 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 02:10:42.557466 8271 catalog_manager.cc:1347] Generated new cluster ID: 0e2304dfbb9940b1a0a2cc90d836f0a2
I20260928 02:10:42.557595 8271 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 02:10:42.566871 8271 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 02:10:42.568616 8271 catalog_manager.cc:1504] Loading token signing keys...
I20260928 02:10:42.579926 8271 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ae6ca031c9744a50b87bd00e3e3df444: Generated new TSK 0
I20260928 02:10:42.581152 8271 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.6.235.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.235.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.235.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790561432251112-7085-0/krb5kdc/kudu_127.6.235.65.keytab.
Entry for principal kudu/127.6.235.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790561432251112-7085-0/krb5kdc/kudu_127.6.235.65.keytab.
I20260928 02:10:42.637662 7085 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTlaUuc/build/asan/bin/kudu
/tmp/dist-test-taskTlaUuc/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790561432251112-7085-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790561432251112-7085-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790561432251112-7085-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790561432251112-7085-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.235.65:0
--local_ip_for_outbound_sockets=127.6.235.65
--webserver_interface=127.6.235.65
--webserver_port=0
--tserver_master_addrs=127.6.235.126:37431
--keytab_file=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790561432251112-7085-0/krb5kdc/kudu_127.6.235.65.keytab
--principal=kudu/127.6.235.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.235.84:34167
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790561432251112-7085-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790561432251112-7085-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790561432251112-7085-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:10:42.983443 8283 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:10:42.983754 8283 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:10:42.983934 8283 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:10:42.991601 8283 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:10:42.991726 8283 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:10:42.991868 8283 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260928 02:10:42.992012 8283 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.235.65
I20260928 02:10:43.002362 8283 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.235.84:34167
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790561432251112-7085-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790561432251112-7085-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790561432251112-7085-0/krb5kdc/kudu_127.6.235.65.keytab
--principal=kudu/127.6.235.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.235.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790561432251112-7085-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.6.235.65
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.6.235.126:37431
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.235.65
--log_dir=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790561432251112-7085-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 00:51:59 UTC on ccycloud.quasar-axgkwn.root.comops.site
build id 4676464
ASAN enabled
I20260928 02:10:43.003779 8283 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:10:43.005306 8283 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:10:43.020223 8291 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:10:43.020265 8289 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:10:43.020287 8288 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:10:43.021847 8283 server_base.cc:1034] running on GCE node
I20260928 02:10:43.022657 8283 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:10:43.023759 8283 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:10:43.025080 8283 hybrid_clock.cc:648] HybridClock initialized: now 1790561443024970 us; error 69 us; skew 500 ppm
I20260928 02:10:43.025439 8283 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 02:10:43 dist-test-slave-86gv krb5kdc[8167](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561443, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.235.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:10:43.029209 8283 init.cc:377] Logged in from keytab as kudu/127.6.235.65@KRBTEST.COM (short username kudu)
I20260928 02:10:43.030913 8283 webserver.cc:460] Webserver started at http://127.6.235.65:37659/ using document root <none> and password file <none>
I20260928 02:10:43.031675 8283 fs_manager.cc:359] Metadata directory not provided
I20260928 02:10:43.031810 8283 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:10:43.032156 8283 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:10:43.035269 8283 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790561432251112-7085-0/minicluster-data/ts-0/data/instance:
uuid: "1296711f54774aa2880bc8d1d2ec7b70"
format_stamp: "Formatted at 2026-09-28 02:10:43 on dist-test-slave-86gv"
server_key: "ea2085c6526c41e31b5c6f347e43061d"
server_key_iv: "c5a4d773a9a427774900f8bfddb29683"
server_key_version: "encryptionkey@0"
I20260928 02:10:43.036140 8283 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790561432251112-7085-0/minicluster-data/ts-0/wal/instance:
uuid: "1296711f54774aa2880bc8d1d2ec7b70"
format_stamp: "Formatted at 2026-09-28 02:10:43 on dist-test-slave-86gv"
server_key: "ea2085c6526c41e31b5c6f347e43061d"
server_key_iv: "c5a4d773a9a427774900f8bfddb29683"
server_key_version: "encryptionkey@0"
I20260928 02:10:43.041889 8283 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260928 02:10:43.045794 8298 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:10:43.047132 8283 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260928 02:10:43.047322 8283 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790561432251112-7085-0/minicluster-data/ts-0/data,/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790561432251112-7085-0/minicluster-data/ts-0/wal
uuid: "1296711f54774aa2880bc8d1d2ec7b70"
format_stamp: "Formatted at 2026-09-28 02:10:43 on dist-test-slave-86gv"
server_key: "ea2085c6526c41e31b5c6f347e43061d"
server_key_iv: "c5a4d773a9a427774900f8bfddb29683"
server_key_version: "encryptionkey@0"
I20260928 02:10:43.047577 8283 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790561432251112-7085-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790561432251112-7085-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790561432251112-7085-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:10:43.074765 8283 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:10:43.079049 8283 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:10:43.079397 8283 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:10:43.082237 8283 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:10:43.082398 8283 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.001s
I20260928 02:10:43.082607 8283 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:10:43.082703 8283 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 28 02:10:43 dist-test-slave-86gv krb5kdc[8167](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561443, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.235.65@KRBTEST.COM for kudu/127.6.235.126@KRBTEST.COM
I20260928 02:10:43.107372 8308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:10:43.083467 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.235.126:37431 (local address 127.6.235.65:34519)
0928 02:10:43.084343 (+ 876us) negotiation.cc:107] Waiting for socket to connect
0928 02:10:43.084384 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0928 02:10:43.086295 (+ 1911us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:10:43.087246 (+ 951us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:10:43.087259 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:10:43.088112 (+ 853us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:10:43.089125 (+ 1013us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:10:43.089163 (+ 38us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:10:43.090576 (+ 1413us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:10:43.090582 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:10:43.092876 (+ 2294us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:10:43.092925 (+ 49us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:10:43.093897 (+ 972us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:10:43.093904 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:10:43.094141 (+ 237us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:10:43.094843 (+ 702us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:10:43.094865 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:10:43.098015 (+ 3150us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:10:43.101750 (+ 3735us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:10:43.101771 (+ 21us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:10:43.101789 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:10:43.102648 (+ 859us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:10:43.103082 (+ 434us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:10:43.103088 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:10:43.103091 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:10:43.103425 (+ 334us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:10:43.104169 (+ 744us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:10:43.104178 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:10:43.104747 (+ 569us) client_negotiation.cc:769] Sending connection context
0928 02:10:43.105139 (+ 392us) client_negotiation.cc:240] Negotiation successful
0928 02:10:43.105748 (+ 609us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":602,"thread_start_us":325,"threads_started":1}
I20260928 02:10:43.108336 8306 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:10:43.082725 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.235.65:34519 (local address 127.6.235.126:37431)
0928 02:10:43.083174 (+ 449us) server_negotiation.cc:206] Beginning negotiation
0928 02:10:43.083183 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 02:10:43.084501 (+ 1318us) server_negotiation.cc:407] Connection header received
0928 02:10:43.086672 (+ 2171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:10:43.086680 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:10:43.086778 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:10:43.086910 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:10:43.089382 (+ 2472us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:10:43.090424 (+ 1042us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:10:43.093099 (+ 2675us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:10:43.093651 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:10:43.093784 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:10:43.098282 (+ 4498us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:10:43.098287 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:10:43.098290 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:10:43.098322 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:10:43.101576 (+ 3254us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:10:43.102808 (+ 1232us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:10:43.102816 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:10:43.102821 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:10:43.102957 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:10:43.103544 (+ 587us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:10:43.103549 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:10:43.103553 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:10:43.103936 (+ 383us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:10:43.104092 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0928 02:10:43.107947 (+ 3855us) server_negotiation.cc:299] Negotiation successful
0928 02:10:43.108116 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":315,"thread_start_us":235,"threads_started":1}
I20260928 02:10:43.130726 8306 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:10:43.117066 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.235.65:34045 (local address 127.6.235.126:37431)
0928 02:10:43.117225 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0928 02:10:43.117230 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 02:10:43.117861 (+ 631us) server_negotiation.cc:407] Connection header received
0928 02:10:43.118091 (+ 230us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:10:43.118095 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:10:43.118158 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:10:43.118261 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:10:43.119183 (+ 922us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:10:43.119871 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:10:43.121311 (+ 1440us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:10:43.121882 (+ 571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:10:43.122014 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:10:43.124200 (+ 2186us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:10:43.124207 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:10:43.124211 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:10:43.124251 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:10:43.127624 (+ 3373us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:10:43.128566 (+ 942us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:10:43.128573 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:10:43.128577 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:10:43.128706 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:10:43.129131 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:10:43.129139 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:10:43.129143 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:10:43.129551 (+ 408us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:10:43.129689 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0928 02:10:43.130130 (+ 441us) server_negotiation.cc:299] Negotiation successful
0928 02:10:43.130355 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260928 02:10:43.130899 8389 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:10:43.117164 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.235.126:37431 (local address 127.6.235.65:34045)
0928 02:10:43.117749 (+ 585us) negotiation.cc:107] Waiting for socket to connect
0928 02:10:43.117764 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 02:10:43.117930 (+ 166us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:10:43.118284 (+ 354us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:10:43.118288 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:10:43.118758 (+ 470us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:10:43.119041 (+ 283us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:10:43.119050 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:10:43.120007 (+ 957us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:10:43.120012 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:10:43.121155 (+ 1143us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:10:43.121167 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:10:43.122266 (+ 1099us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:10:43.122273 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:10:43.122507 (+ 234us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:10:43.123193 (+ 686us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:10:43.123214 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:10:43.124000 (+ 786us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:10:43.127761 (+ 3761us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:10:43.127768 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:10:43.127776 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:10:43.128418 (+ 642us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:10:43.128818 (+ 400us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:10:43.128823 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:10:43.128830 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:10:43.129024 (+ 194us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:10:43.129655 (+ 631us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:10:43.129660 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:10:43.129866 (+ 206us) client_negotiation.cc:769] Sending connection context
0928 02:10:43.130013 (+ 147us) client_negotiation.cc:240] Negotiation successful
0928 02:10:43.130371 (+ 358us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":466,"thread_start_us":330,"threads_started":1}
I20260928 02:10:43.132256 8283 rpc_server.cc:307] RPC server started. Bound to: 127.6.235.65:34341
I20260928 02:10:43.132294 8418 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.235.65:34341 every 8 connection(s)
I20260928 02:10:43.133651 8283 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790561432251112-7085-0/minicluster-data/ts-0/data/info.pb
I20260928 02:10:43.135905 8419 heartbeater.cc:347] Connected to a master server at 127.6.235.126:37431
I20260928 02:10:43.136127 8419 heartbeater.cc:464] Registering TS with master...
I20260928 02:10:43.136721 8419 heartbeater.cc:511] Master 127.6.235.126:37431 requested a full tablet report, sending...
I20260928 02:10:43.138716 8212 ts_manager.cc:194] Registered new tserver with Master: 1296711f54774aa2880bc8d1d2ec7b70 (127.6.235.65:34341)
I20260928 02:10:43.140136 8212 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.235.65@KRBTEST.COM'} at 127.6.235.65:34519
I20260928 02:10:43.143226 7085 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTlaUuc/build/asan/bin/kudu as pid 8283
I20260928 02:10:43.143399 7085 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790561432251112-7085-0/minicluster-data/ts-0/wal/instance
I20260928 02:10:43.143946 7085 external_mini_cluster.cc:1428] Setting key c00aafec78466bc93176451e54692c37
WARNING: no policy specified for kudu/127.6.235.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.235.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.235.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790561432251112-7085-0/krb5kdc/kudu_127.6.235.66.keytab.
Entry for principal kudu/127.6.235.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790561432251112-7085-0/krb5kdc/kudu_127.6.235.66.keytab.
I20260928 02:10:43.188562 7085 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTlaUuc/build/asan/bin/kudu
/tmp/dist-test-taskTlaUuc/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790561432251112-7085-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790561432251112-7085-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790561432251112-7085-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790561432251112-7085-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.235.66:0
--local_ip_for_outbound_sockets=127.6.235.66
--webserver_interface=127.6.235.66
--webserver_port=0
--tserver_master_addrs=127.6.235.126:37431
--keytab_file=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790561432251112-7085-0/krb5kdc/kudu_127.6.235.66.keytab
--principal=kudu/127.6.235.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.235.84:34167
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790561432251112-7085-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790561432251112-7085-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790561432251112-7085-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:10:43.526548 8423 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:10:43.526887 8423 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:10:43.527062 8423 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:10:43.534461 8423 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:10:43.534629 8423 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:10:43.534772 8423 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260928 02:10:43.534947 8423 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.235.66
I20260928 02:10:43.545480 8423 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.235.84:34167
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790561432251112-7085-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790561432251112-7085-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790561432251112-7085-0/krb5kdc/kudu_127.6.235.66.keytab
--principal=kudu/127.6.235.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.235.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790561432251112-7085-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.6.235.66
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.6.235.126:37431
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.235.66
--log_dir=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790561432251112-7085-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 00:51:59 UTC on ccycloud.quasar-axgkwn.root.comops.site
build id 4676464
ASAN enabled
I20260928 02:10:43.546923 8423 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:10:43.548345 8423 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:10:43.562683 8428 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:10:43.562709 8431 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:10:43.562625 8429 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:10:43.563720 8423 server_base.cc:1034] running on GCE node
I20260928 02:10:43.564411 8423 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:10:43.565457 8423 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:10:43.566689 8423 hybrid_clock.cc:648] HybridClock initialized: now 1790561443566622 us; error 37 us; skew 500 ppm
I20260928 02:10:43.567021 8423 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 02:10:43 dist-test-slave-86gv krb5kdc[8167](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561443, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.235.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:10:43.570472 8423 init.cc:377] Logged in from keytab as kudu/127.6.235.66@KRBTEST.COM (short username kudu)
I20260928 02:10:43.572219 8423 webserver.cc:460] Webserver started at http://127.6.235.66:45141/ using document root <none> and password file <none>
I20260928 02:10:43.573019 8423 fs_manager.cc:359] Metadata directory not provided
I20260928 02:10:43.573165 8423 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:10:43.573495 8423 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:10:43.576527 8423 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790561432251112-7085-0/minicluster-data/ts-1/data/instance:
uuid: "931a82ffd0ef48a48003604170d99a22"
format_stamp: "Formatted at 2026-09-28 02:10:43 on dist-test-slave-86gv"
server_key: "5dd5a9172ef8dce232f887e0d0d4ac8c"
server_key_iv: "7aa340f0ee2653279f3b37fcbdac42c4"
server_key_version: "encryptionkey@0"
I20260928 02:10:43.577390 8423 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790561432251112-7085-0/minicluster-data/ts-1/wal/instance:
uuid: "931a82ffd0ef48a48003604170d99a22"
format_stamp: "Formatted at 2026-09-28 02:10:43 on dist-test-slave-86gv"
server_key: "5dd5a9172ef8dce232f887e0d0d4ac8c"
server_key_iv: "7aa340f0ee2653279f3b37fcbdac42c4"
server_key_version: "encryptionkey@0"
I20260928 02:10:43.582665 8423 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260928 02:10:43.586416 8438 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:10:43.587972 8423 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.004s
I20260928 02:10:43.588155 8423 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790561432251112-7085-0/minicluster-data/ts-1/data,/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790561432251112-7085-0/minicluster-data/ts-1/wal
uuid: "931a82ffd0ef48a48003604170d99a22"
format_stamp: "Formatted at 2026-09-28 02:10:43 on dist-test-slave-86gv"
server_key: "5dd5a9172ef8dce232f887e0d0d4ac8c"
server_key_iv: "7aa340f0ee2653279f3b37fcbdac42c4"
server_key_version: "encryptionkey@0"
I20260928 02:10:43.588382 8423 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790561432251112-7085-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790561432251112-7085-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790561432251112-7085-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:10:43.601691 8423 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:10:43.605839 8423 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:10:43.606148 8423 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:10:43.609103 8423 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:10:43.609205 8423 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:10:43.609350 8423 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:10:43.609424 8423 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 28 02:10:43 dist-test-slave-86gv krb5kdc[8167](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561443, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.235.66@KRBTEST.COM for kudu/127.6.235.126@KRBTEST.COM
I20260928 02:10:43.633481 8306 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:10:43.609284 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.235.66:58351 (local address 127.6.235.126:37431)
0928 02:10:43.609501 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0928 02:10:43.609507 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 02:10:43.611049 (+ 1542us) server_negotiation.cc:407] Connection header received
0928 02:10:43.613140 (+ 2091us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:10:43.613145 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:10:43.613238 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:10:43.613378 (+ 140us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:10:43.615459 (+ 2081us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:10:43.616824 (+ 1365us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:10:43.619400 (+ 2576us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:10:43.620196 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:10:43.620347 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:10:43.625336 (+ 4989us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:10:43.625347 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:10:43.625354 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:10:43.625410 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:10:43.629094 (+ 3684us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:10:43.630206 (+ 1112us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:10:43.630213 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:10:43.630218 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:10:43.630372 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:10:43.630974 (+ 602us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:10:43.630981 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:10:43.630985 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:10:43.631378 (+ 393us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:10:43.631566 (+ 188us) server_negotiation.cc:1035] Waiting for connection context
0928 02:10:43.632718 (+ 1152us) server_negotiation.cc:299] Negotiation successful
0928 02:10:43.633066 (+ 348us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260928 02:10:43.635386 8447 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:10:43.609963 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.235.126:37431 (local address 127.6.235.66:58351)
0928 02:10:43.610860 (+ 897us) negotiation.cc:107] Waiting for socket to connect
0928 02:10:43.610913 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0928 02:10:43.612765 (+ 1852us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:10:43.613694 (+ 929us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:10:43.613711 (+ 17us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:10:43.614639 (+ 928us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:10:43.615260 (+ 621us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:10:43.615278 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:10:43.617033 (+ 1755us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:10:43.617039 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:10:43.619155 (+ 2116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:10:43.619227 (+ 72us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:10:43.620489 (+ 1262us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:10:43.620496 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:10:43.620694 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:10:43.621549 (+ 855us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:10:43.621591 (+ 42us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:10:43.625039 (+ 3448us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:10:43.629290 (+ 4251us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:10:43.629297 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:10:43.629313 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:10:43.629986 (+ 673us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:10:43.630508 (+ 522us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:10:43.630513 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:10:43.630517 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:10:43.630854 (+ 337us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:10:43.631554 (+ 700us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:10:43.631567 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:10:43.632223 (+ 656us) client_negotiation.cc:769] Sending connection context
0928 02:10:43.632638 (+ 415us) client_negotiation.cc:240] Negotiation successful
0928 02:10:43.633433 (+ 795us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":618,"thread_start_us":365,"threads_started":1}
I20260928 02:10:43.657359 8510 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:10:43.641531 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.235.126:37431 (local address 127.6.235.66:45061)
0928 02:10:43.642206 (+ 675us) negotiation.cc:107] Waiting for socket to connect
0928 02:10:43.642224 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0928 02:10:43.642345 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:10:43.642713 (+ 368us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:10:43.642716 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:10:43.643142 (+ 426us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:10:43.643426 (+ 284us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:10:43.643434 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:10:43.644475 (+ 1041us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:10:43.644482 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:10:43.646066 (+ 1584us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:10:43.646083 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:10:43.647344 (+ 1261us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:10:43.647351 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:10:43.647563 (+ 212us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:10:43.648370 (+ 807us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:10:43.648393 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:10:43.649686 (+ 1293us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:10:43.654319 (+ 4633us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:10:43.654326 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:10:43.654330 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:10:43.654945 (+ 615us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:10:43.655355 (+ 410us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:10:43.655360 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:10:43.655364 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:10:43.655499 (+ 135us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:10:43.656074 (+ 575us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:10:43.656080 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:10:43.656361 (+ 281us) client_negotiation.cc:769] Sending connection context
0928 02:10:43.656552 (+ 191us) client_negotiation.cc:240] Negotiation successful
0928 02:10:43.656920 (+ 368us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":524,"thread_start_us":358,"threads_started":1}
I20260928 02:10:43.657860 8306 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:10:43.641553 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.235.66:45061 (local address 127.6.235.126:37431)
0928 02:10:43.641783 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0928 02:10:43.641792 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 02:10:43.642311 (+ 519us) server_negotiation.cc:407] Connection header received
0928 02:10:43.642446 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:10:43.642454 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:10:43.642555 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:10:43.642678 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:10:43.643576 (+ 898us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:10:43.644296 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:10:43.646297 (+ 2001us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:10:43.647123 (+ 826us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:10:43.647293 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:10:43.649945 (+ 2652us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:10:43.649955 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:10:43.649984 (+ 29us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:10:43.650041 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:10:43.654198 (+ 4157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:10:43.655087 (+ 889us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:10:43.655095 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:10:43.655100 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:10:43.655242 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:10:43.655613 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:10:43.655619 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:10:43.655623 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:10:43.655947 (+ 324us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:10:43.656101 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0928 02:10:43.657107 (+ 1006us) server_negotiation.cc:299] Negotiation successful
0928 02:10:43.657404 (+ 297us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260928 02:10:43.667268 8423 rpc_server.cc:307] RPC server started. Bound to: 127.6.235.66:40407
I20260928 02:10:43.667307 8557 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.235.66:40407 every 8 connection(s)
I20260928 02:10:43.668506 8423 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790561432251112-7085-0/minicluster-data/ts-1/data/info.pb
I20260928 02:10:43.668815 7085 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTlaUuc/build/asan/bin/kudu as pid 8423
I20260928 02:10:43.669057 7085 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790561432251112-7085-0/minicluster-data/ts-1/wal/instance
I20260928 02:10:43.669667 7085 external_mini_cluster.cc:1428] Setting key 77ff833d04d2f6c818d2adcafafe86a6
I20260928 02:10:43.670723 8558 heartbeater.cc:347] Connected to a master server at 127.6.235.126:37431
I20260928 02:10:43.670969 8558 heartbeater.cc:464] Registering TS with master...
I20260928 02:10:43.671741 8558 heartbeater.cc:511] Master 127.6.235.126:37431 requested a full tablet report, sending...
I20260928 02:10:43.673426 8212 ts_manager.cc:194] Registered new tserver with Master: 931a82ffd0ef48a48003604170d99a22 (127.6.235.66:40407)
I20260928 02:10:43.674731 8212 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.235.66@KRBTEST.COM'} at 127.6.235.66:58351
WARNING: no policy specified for kudu/127.6.235.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.235.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.235.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790561432251112-7085-0/krb5kdc/kudu_127.6.235.67.keytab.
Entry for principal kudu/127.6.235.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790561432251112-7085-0/krb5kdc/kudu_127.6.235.67.keytab.
I20260928 02:10:43.714650 7085 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTlaUuc/build/asan/bin/kudu
/tmp/dist-test-taskTlaUuc/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790561432251112-7085-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790561432251112-7085-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790561432251112-7085-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790561432251112-7085-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.235.67:0
--local_ip_for_outbound_sockets=127.6.235.67
--webserver_interface=127.6.235.67
--webserver_port=0
--tserver_master_addrs=127.6.235.126:37431
--keytab_file=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790561432251112-7085-0/krb5kdc/kudu_127.6.235.67.keytab
--principal=kudu/127.6.235.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.235.84:34167
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790561432251112-7085-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790561432251112-7085-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790561432251112-7085-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:10:44.050674 8562 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:10:44.051009 8562 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:10:44.051182 8562 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:10:44.058476 8562 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:10:44.058656 8562 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:10:44.058817 8562 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260928 02:10:44.058959 8562 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.235.67
I20260928 02:10:44.068804 8562 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.235.84:34167
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790561432251112-7085-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790561432251112-7085-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790561432251112-7085-0/krb5kdc/kudu_127.6.235.67.keytab
--principal=kudu/127.6.235.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.235.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790561432251112-7085-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.6.235.67
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.6.235.126:37431
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.235.67
--log_dir=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790561432251112-7085-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 00:51:59 UTC on ccycloud.quasar-axgkwn.root.comops.site
build id 4676464
ASAN enabled
I20260928 02:10:44.070210 8562 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:10:44.071643 8562 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:10:44.085081 8567 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:10:44.085161 8570 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:10:44.085124 8568 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:10:44.086135 8562 server_base.cc:1034] running on GCE node
I20260928 02:10:44.086768 8562 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:10:44.087769 8562 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:10:44.089033 8562 hybrid_clock.cc:648] HybridClock initialized: now 1790561444088928 us; error 54 us; skew 500 ppm
I20260928 02:10:44.089354 8562 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 02:10:44 dist-test-slave-86gv krb5kdc[8167](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561444, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.235.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:10:44.092599 8562 init.cc:377] Logged in from keytab as kudu/127.6.235.67@KRBTEST.COM (short username kudu)
I20260928 02:10:44.094204 8562 webserver.cc:460] Webserver started at http://127.6.235.67:45137/ using document root <none> and password file <none>
I20260928 02:10:44.094949 8562 fs_manager.cc:359] Metadata directory not provided
I20260928 02:10:44.095089 8562 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:10:44.095407 8562 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:10:44.098853 8562 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790561432251112-7085-0/minicluster-data/ts-2/data/instance:
uuid: "e84d3c7d475a4ad9a1d40d8938b60cf7"
format_stamp: "Formatted at 2026-09-28 02:10:44 on dist-test-slave-86gv"
server_key: "c0dd0c6a0aa2562576daed94ef5c5096"
server_key_iv: "10e185e9b577e6b26d6109016c207f8a"
server_key_version: "encryptionkey@0"
I20260928 02:10:44.099700 8562 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790561432251112-7085-0/minicluster-data/ts-2/wal/instance:
uuid: "e84d3c7d475a4ad9a1d40d8938b60cf7"
format_stamp: "Formatted at 2026-09-28 02:10:44 on dist-test-slave-86gv"
server_key: "c0dd0c6a0aa2562576daed94ef5c5096"
server_key_iv: "10e185e9b577e6b26d6109016c207f8a"
server_key_version: "encryptionkey@0"
I20260928 02:10:44.105391 8562 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.007s
I20260928 02:10:44.108964 8577 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:10:44.110445 8562 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260928 02:10:44.110632 8562 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790561432251112-7085-0/minicluster-data/ts-2/data,/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790561432251112-7085-0/minicluster-data/ts-2/wal
uuid: "e84d3c7d475a4ad9a1d40d8938b60cf7"
format_stamp: "Formatted at 2026-09-28 02:10:44 on dist-test-slave-86gv"
server_key: "c0dd0c6a0aa2562576daed94ef5c5096"
server_key_iv: "10e185e9b577e6b26d6109016c207f8a"
server_key_version: "encryptionkey@0"
I20260928 02:10:44.110855 8562 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790561432251112-7085-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790561432251112-7085-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790561432251112-7085-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:10:44.126312 8562 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:10:44.130630 8562 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:10:44.131026 8562 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:10:44.133924 8562 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:10:44.134042 8562 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:10:44.134198 8562 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:10:44.134275 8562 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:10:44.143028 8419 heartbeater.cc:503] Master 127.6.235.126:37431 was elected leader, sending a full tablet report...
Sep 28 02:10:44 dist-test-slave-86gv krb5kdc[8167](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561444, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.235.67@KRBTEST.COM for kudu/127.6.235.126@KRBTEST.COM
I20260928 02:10:44.161463 8306 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:10:44.134707 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.235.67:59713 (local address 127.6.235.126:37431)
0928 02:10:44.134888 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0928 02:10:44.134895 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 02:10:44.136518 (+ 1623us) server_negotiation.cc:407] Connection header received
0928 02:10:44.139226 (+ 2708us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:10:44.139237 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:10:44.139347 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:10:44.139466 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:10:44.141936 (+ 2470us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:10:44.143014 (+ 1078us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:10:44.146019 (+ 3005us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:10:44.146659 (+ 640us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:10:44.146824 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:10:44.152637 (+ 5813us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:10:44.152643 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:10:44.152646 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:10:44.152680 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:10:44.156816 (+ 4136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:10:44.158101 (+ 1285us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:10:44.158109 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:10:44.158114 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:10:44.158257 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:10:44.158943 (+ 686us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:10:44.158949 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:10:44.158953 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:10:44.159255 (+ 302us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:10:44.159370 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0928 02:10:44.160589 (+ 1219us) server_negotiation.cc:299] Negotiation successful
0928 02:10:44.160926 (+ 337us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260928 02:10:44.163545 8586 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:10:44.135424 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.235.126:37431 (local address 127.6.235.67:59713)
0928 02:10:44.136336 (+ 912us) negotiation.cc:107] Waiting for socket to connect
0928 02:10:44.136382 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0928 02:10:44.138908 (+ 2526us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:10:44.139738 (+ 830us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:10:44.139753 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:10:44.140508 (+ 755us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:10:44.141711 (+ 1203us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:10:44.141745 (+ 34us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:10:44.143174 (+ 1429us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:10:44.143183 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:10:44.145855 (+ 2672us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:10:44.145873 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:10:44.146941 (+ 1068us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:10:44.146949 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:10:44.147238 (+ 289us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:10:44.148218 (+ 980us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:10:44.148260 (+ 42us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:10:44.152329 (+ 4069us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:10:44.157011 (+ 4682us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:10:44.157026 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:10:44.157055 (+ 29us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:10:44.157952 (+ 897us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:10:44.158379 (+ 427us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:10:44.158387 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:10:44.158394 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:10:44.158777 (+ 383us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:10:44.159406 (+ 629us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:10:44.159422 (+ 16us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:10:44.160040 (+ 618us) client_negotiation.cc:769] Sending connection context
0928 02:10:44.160445 (+ 405us) client_negotiation.cc:240] Negotiation successful
0928 02:10:44.161280 (+ 835us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":599,"thread_start_us":612,"threads_started":1}
I20260928 02:10:44.180581 8562 rpc_server.cc:307] RPC server started. Bound to: 127.6.235.67:42213
I20260928 02:10:44.180688 8696 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.235.67:42213 every 8 connection(s)
I20260928 02:10:44.182428 8562 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790561432251112-7085-0/minicluster-data/ts-2/data/info.pb
I20260928 02:10:44.183740 7085 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTlaUuc/build/asan/bin/kudu as pid 8562
I20260928 02:10:44.183948 7085 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790561432251112-7085-0/minicluster-data/ts-2/wal/instance
I20260928 02:10:44.184445 7085 external_mini_cluster.cc:1428] Setting key eaf7264020887c0f5cf0c7bec5767abc
I20260928 02:10:44.185096 8697 heartbeater.cc:347] Connected to a master server at 127.6.235.126:37431
I20260928 02:10:44.185335 8697 heartbeater.cc:464] Registering TS with master...
I20260928 02:10:44.186199 8697 heartbeater.cc:511] Master 127.6.235.126:37431 requested a full tablet report, sending...
I20260928 02:10:44.188069 8211 ts_manager.cc:194] Registered new tserver with Master: e84d3c7d475a4ad9a1d40d8938b60cf7 (127.6.235.67:42213)
I20260928 02:10:44.189486 8211 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.235.67@KRBTEST.COM'} at 127.6.235.67:59713
I20260928 02:10:44.192044 8306 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:10:44.172669 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.235.67:51047 (local address 127.6.235.126:37431)
0928 02:10:44.172933 (+ 264us) server_negotiation.cc:206] Beginning negotiation
0928 02:10:44.172942 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 02:10:44.174535 (+ 1593us) server_negotiation.cc:407] Connection header received
0928 02:10:44.174818 (+ 283us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:10:44.174826 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:10:44.174934 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:10:44.175052 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:10:44.176809 (+ 1757us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:10:44.178126 (+ 1317us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:10:44.180411 (+ 2285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:10:44.181129 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:10:44.181245 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:10:44.185348 (+ 4103us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:10:44.185356 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:10:44.185361 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:10:44.185408 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:10:44.189383 (+ 3975us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:10:44.190218 (+ 835us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:10:44.190222 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:10:44.190226 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:10:44.190386 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:10:44.190766 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:10:44.190769 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:10:44.190771 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:10:44.191013 (+ 242us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:10:44.191115 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0928 02:10:44.191647 (+ 532us) server_negotiation.cc:299] Negotiation successful
0928 02:10:44.191806 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260928 02:10:44.192243 8676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:10:44.172671 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.235.126:37431 (local address 127.6.235.67:51047)
0928 02:10:44.174363 (+ 1692us) negotiation.cc:107] Waiting for socket to connect
0928 02:10:44.174390 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0928 02:10:44.174631 (+ 241us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:10:44.175101 (+ 470us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:10:44.175109 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:10:44.175993 (+ 884us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:10:44.176564 (+ 571us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:10:44.176584 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:10:44.178350 (+ 1766us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:10:44.178355 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:10:44.180182 (+ 1827us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:10:44.180206 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:10:44.181337 (+ 1131us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:10:44.181344 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:10:44.181539 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:10:44.182236 (+ 697us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:10:44.182259 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:10:44.185154 (+ 2895us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:10:44.189541 (+ 4387us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:10:44.189546 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:10:44.189549 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:10:44.190096 (+ 547us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:10:44.190503 (+ 407us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:10:44.190509 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:10:44.190512 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:10:44.190653 (+ 141us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:10:44.191140 (+ 487us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:10:44.191146 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:10:44.191322 (+ 176us) client_negotiation.cc:769] Sending connection context
0928 02:10:44.191476 (+ 154us) client_negotiation.cc:240] Negotiation successful
0928 02:10:44.191783 (+ 307us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1462,"thread_start_us":1125,"threads_started":1}
I20260928 02:10:44.200989 7085 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 28 02:10:44 dist-test-slave-86gv krb5kdc[8167](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561444, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 28 02:10:44 dist-test-slave-86gv krb5kdc[8167](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561444, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.6.235.126@KRBTEST.COM
I20260928 02:10:44.257249 8306 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:10:44.234688 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46612 (local address 127.6.235.126:37431)
0928 02:10:44.234918 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0928 02:10:44.234927 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 02:10:44.235220 (+ 293us) server_negotiation.cc:407] Connection header received
0928 02:10:44.235426 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:10:44.235432 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:10:44.235504 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:10:44.235613 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:10:44.237060 (+ 1447us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:10:44.237884 (+ 824us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:10:44.239190 (+ 1306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:10:44.239654 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:10:44.239750 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:10:44.251555 (+ 11805us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:10:44.251561 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:10:44.251565 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:10:44.251611 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:10:44.254396 (+ 2785us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:10:44.255172 (+ 776us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:10:44.255179 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:10:44.255182 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:10:44.255322 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:10:44.255759 (+ 437us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:10:44.255764 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:10:44.255766 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:10:44.256059 (+ 293us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:10:44.256312 (+ 253us) server_negotiation.cc:1035] Waiting for connection context
0928 02:10:44.256637 (+ 325us) server_negotiation.cc:299] Negotiation successful
0928 02:10:44.256854 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260928 02:10:44.261612 8211 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:46612:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 02:10:44.268829 8211 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 02:10:44.307224 8713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:10:44.283761 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45692 (local address 127.6.235.67:42213)
0928 02:10:44.295088 (+ 11327us) server_negotiation.cc:206] Beginning negotiation
0928 02:10:44.295103 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0928 02:10:44.298028 (+ 2925us) server_negotiation.cc:407] Connection header received
0928 02:10:44.298304 (+ 276us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:10:44.298311 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:10:44.298505 (+ 194us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:10:44.298687 (+ 182us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:10:44.300265 (+ 1578us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:10:44.301920 (+ 1655us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:10:44.304909 (+ 2989us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:10:44.306011 (+ 1102us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:10:44.306166 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:10:44.306439 (+ 273us) server_negotiation.cc:1035] Waiting for connection context
0928 02:10:44.306703 (+ 264us) server_negotiation.cc:299] Negotiation successful
0928 02:10:44.306868 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11077,"thread_start_us":10837,"threads_started":1}
I20260928 02:10:44.307462 8710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:10:44.282132 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.235.67:42213 (local address 127.0.0.1:45692)
0928 02:10:44.297701 (+ 15569us) negotiation.cc:107] Waiting for socket to connect
0928 02:10:44.297721 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0928 02:10:44.297869 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:10:44.298981 (+ 1112us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:10:44.298984 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:10:44.299020 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:10:44.300023 (+ 1003us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:10:44.300033 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:10:44.302064 (+ 2031us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:10:44.302074 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:10:44.304508 (+ 2434us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:10:44.304526 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:10:44.306222 (+ 1696us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:10:44.306228 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:10:44.306416 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:10:44.306476 (+ 60us) client_negotiation.cc:769] Sending connection context
0928 02:10:44.306650 (+ 174us) client_negotiation.cc:240] Negotiation successful
0928 02:10:44.306856 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15408,"mutex_wait_us":322,"spinlock_wait_cycles":68096,"thread_start_us":12990,"threads_started":1}
I20260928 02:10:44.308420 8715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:10:44.296325 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34854 (local address 127.6.235.66:40407)
0928 02:10:44.297410 (+ 1085us) server_negotiation.cc:206] Beginning negotiation
0928 02:10:44.297419 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 02:10:44.297906 (+ 487us) server_negotiation.cc:407] Connection header received
0928 02:10:44.298201 (+ 295us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:10:44.298211 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:10:44.298505 (+ 294us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:10:44.298685 (+ 180us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:10:44.300196 (+ 1511us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:10:44.301799 (+ 1603us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:10:44.305189 (+ 3390us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:10:44.306629 (+ 1440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:10:44.306769 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:10:44.306971 (+ 202us) server_negotiation.cc:1035] Waiting for connection context
0928 02:10:44.307974 (+ 1003us) server_negotiation.cc:299] Negotiation successful
0928 02:10:44.308151 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":921,"thread_start_us":935,"threads_started":1}
I20260928 02:10:44.309177 8714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:10:44.296294 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.235.66:40407 (local address 127.0.0.1:34854)
0928 02:10:44.297810 (+ 1516us) negotiation.cc:107] Waiting for socket to connect
0928 02:10:44.297824 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 02:10:44.298011 (+ 187us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:10:44.298958 (+ 947us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:10:44.298972 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:10:44.299029 (+ 57us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:10:44.299928 (+ 899us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:10:44.299946 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:10:44.301937 (+ 1991us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:10:44.301944 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:10:44.304967 (+ 3023us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:10:44.304985 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:10:44.307155 (+ 2170us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:10:44.307163 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:10:44.307497 (+ 334us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:10:44.307529 (+ 32us) client_negotiation.cc:769] Sending connection context
0928 02:10:44.307626 (+ 97us) client_negotiation.cc:240] Negotiation successful
0928 02:10:44.307752 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1403,"mutex_wait_us":331,"spinlock_wait_cycles":768,"thread_start_us":1276,"threads_started":1}
I20260928 02:10:44.309554 8711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:10:44.281791 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.235.65:34341 (local address 127.0.0.1:47860)
0928 02:10:44.295568 (+ 13777us) negotiation.cc:107] Waiting for socket to connect
0928 02:10:44.295616 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0928 02:10:44.295844 (+ 228us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:10:44.298979 (+ 3135us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:10:44.298986 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:10:44.299071 (+ 85us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:10:44.300228 (+ 1157us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:10:44.300239 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:10:44.301776 (+ 1537us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:10:44.301782 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:10:44.304914 (+ 3132us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:10:44.304926 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:10:44.307170 (+ 2244us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:10:44.307175 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:10:44.308726 (+ 1551us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:10:44.308759 (+ 33us) client_negotiation.cc:769] Sending connection context
0928 02:10:44.308874 (+ 115us) client_negotiation.cc:240] Negotiation successful
0928 02:10:44.309040 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13477,"mutex_wait_us":281,"spinlock_wait_cycles":14848,"thread_start_us":14227,"threads_started":1}
I20260928 02:10:44.309871 8712 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:10:44.282054 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47860 (local address 127.6.235.65:34341)
0928 02:10:44.295619 (+ 13565us) server_negotiation.cc:206] Beginning negotiation
0928 02:10:44.295630 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 02:10:44.297876 (+ 2246us) server_negotiation.cc:407] Connection header received
0928 02:10:44.298233 (+ 357us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:10:44.298242 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:10:44.298632 (+ 390us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:10:44.299394 (+ 762us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:10:44.300426 (+ 1032us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:10:44.301632 (+ 1206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:10:44.305068 (+ 3436us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:10:44.306629 (+ 1561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:10:44.306769 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:10:44.306975 (+ 206us) server_negotiation.cc:1035] Waiting for connection context
0928 02:10:44.308953 (+ 1978us) server_negotiation.cc:299] Negotiation successful
0928 02:10:44.309165 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13355,"thread_start_us":13567,"threads_started":1}
I20260928 02:10:44.312572 8488 tablet_service.cc:1467] Processing CreateTablet for tablet 8981328c9d994a0ea7cafbfebe3132b2 (DEFAULT_TABLE table=test-table [id=e40ea3ba5e6949488e406fcb75d0ab59]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:10:44.312692 8627 tablet_service.cc:1467] Processing CreateTablet for tablet 8981328c9d994a0ea7cafbfebe3132b2 (DEFAULT_TABLE table=test-table [id=e40ea3ba5e6949488e406fcb75d0ab59]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:10:44.313293 8349 tablet_service.cc:1467] Processing CreateTablet for tablet 8981328c9d994a0ea7cafbfebe3132b2 (DEFAULT_TABLE table=test-table [id=e40ea3ba5e6949488e406fcb75d0ab59]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:10:44.314123 8488 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8981328c9d994a0ea7cafbfebe3132b2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:10:44.314214 8627 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8981328c9d994a0ea7cafbfebe3132b2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:10:44.314584 8349 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8981328c9d994a0ea7cafbfebe3132b2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:10:44.325410 8716 tablet_bootstrap.cc:492] T 8981328c9d994a0ea7cafbfebe3132b2 P 931a82ffd0ef48a48003604170d99a22: Bootstrap starting.
I20260928 02:10:44.329089 8716 tablet_bootstrap.cc:654] T 8981328c9d994a0ea7cafbfebe3132b2 P 931a82ffd0ef48a48003604170d99a22: Neither blocks nor log segments found. Creating new log.
I20260928 02:10:44.329664 8717 tablet_bootstrap.cc:492] T 8981328c9d994a0ea7cafbfebe3132b2 P 1296711f54774aa2880bc8d1d2ec7b70: Bootstrap starting.
I20260928 02:10:44.330660 8716 log.cc:824] T 8981328c9d994a0ea7cafbfebe3132b2 P 931a82ffd0ef48a48003604170d99a22: Log is configured to *not* fsync() on all Append() calls
I20260928 02:10:44.332218 8718 tablet_bootstrap.cc:492] T 8981328c9d994a0ea7cafbfebe3132b2 P e84d3c7d475a4ad9a1d40d8938b60cf7: Bootstrap starting.
I20260928 02:10:44.333702 8717 tablet_bootstrap.cc:654] T 8981328c9d994a0ea7cafbfebe3132b2 P 1296711f54774aa2880bc8d1d2ec7b70: Neither blocks nor log segments found. Creating new log.
I20260928 02:10:44.334580 8716 tablet_bootstrap.cc:492] T 8981328c9d994a0ea7cafbfebe3132b2 P 931a82ffd0ef48a48003604170d99a22: No bootstrap required, opened a new log
I20260928 02:10:44.334867 8716 ts_tablet_manager.cc:1397] T 8981328c9d994a0ea7cafbfebe3132b2 P 931a82ffd0ef48a48003604170d99a22: Time spent bootstrapping tablet: real 0.010s user 0.003s sys 0.003s
I20260928 02:10:44.334981 8717 log.cc:824] T 8981328c9d994a0ea7cafbfebe3132b2 P 1296711f54774aa2880bc8d1d2ec7b70: Log is configured to *not* fsync() on all Append() calls
I20260928 02:10:44.335893 8718 tablet_bootstrap.cc:654] T 8981328c9d994a0ea7cafbfebe3132b2 P e84d3c7d475a4ad9a1d40d8938b60cf7: Neither blocks nor log segments found. Creating new log.
I20260928 02:10:44.337152 8718 log.cc:824] T 8981328c9d994a0ea7cafbfebe3132b2 P e84d3c7d475a4ad9a1d40d8938b60cf7: Log is configured to *not* fsync() on all Append() calls
I20260928 02:10:44.338375 8717 tablet_bootstrap.cc:492] T 8981328c9d994a0ea7cafbfebe3132b2 P 1296711f54774aa2880bc8d1d2ec7b70: No bootstrap required, opened a new log
I20260928 02:10:44.338683 8717 ts_tablet_manager.cc:1397] T 8981328c9d994a0ea7cafbfebe3132b2 P 1296711f54774aa2880bc8d1d2ec7b70: Time spent bootstrapping tablet: real 0.009s user 0.007s sys 0.000s
I20260928 02:10:44.346284 8718 tablet_bootstrap.cc:492] T 8981328c9d994a0ea7cafbfebe3132b2 P e84d3c7d475a4ad9a1d40d8938b60cf7: No bootstrap required, opened a new log
I20260928 02:10:44.345944 8716 raft_consensus.cc:348] T 8981328c9d994a0ea7cafbfebe3132b2 P 931a82ffd0ef48a48003604170d99a22 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1296711f54774aa2880bc8d1d2ec7b70" member_type: VOTER last_known_addr { host: "127.6.235.65" port: 34341 } } peers { permanent_uuid: "e84d3c7d475a4ad9a1d40d8938b60cf7" member_type: VOTER last_known_addr { host: "127.6.235.67" port: 42213 } } peers { permanent_uuid: "931a82ffd0ef48a48003604170d99a22" member_type: VOTER last_known_addr { host: "127.6.235.66" port: 40407 } }
I20260928 02:10:44.346458 8716 raft_consensus.cc:374] T 8981328c9d994a0ea7cafbfebe3132b2 P 931a82ffd0ef48a48003604170d99a22 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:10:44.346561 8716 raft_consensus.cc:729] T 8981328c9d994a0ea7cafbfebe3132b2 P 931a82ffd0ef48a48003604170d99a22 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 931a82ffd0ef48a48003604170d99a22, State: Initialized, Role: FOLLOWER
I20260928 02:10:44.346647 8718 ts_tablet_manager.cc:1397] T 8981328c9d994a0ea7cafbfebe3132b2 P e84d3c7d475a4ad9a1d40d8938b60cf7: Time spent bootstrapping tablet: real 0.015s user 0.004s sys 0.008s
I20260928 02:10:44.347244 8716 consensus_queue.cc:260] T 8981328c9d994a0ea7cafbfebe3132b2 P 931a82ffd0ef48a48003604170d99a22 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1296711f54774aa2880bc8d1d2ec7b70" member_type: VOTER last_known_addr { host: "127.6.235.65" port: 34341 } } peers { permanent_uuid: "e84d3c7d475a4ad9a1d40d8938b60cf7" member_type: VOTER last_known_addr { host: "127.6.235.67" port: 42213 } } peers { permanent_uuid: "931a82ffd0ef48a48003604170d99a22" member_type: VOTER last_known_addr { host: "127.6.235.66" port: 40407 } }
I20260928 02:10:44.349975 8717 raft_consensus.cc:348] T 8981328c9d994a0ea7cafbfebe3132b2 P 1296711f54774aa2880bc8d1d2ec7b70 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1296711f54774aa2880bc8d1d2ec7b70" member_type: VOTER last_known_addr { host: "127.6.235.65" port: 34341 } } peers { permanent_uuid: "e84d3c7d475a4ad9a1d40d8938b60cf7" member_type: VOTER last_known_addr { host: "127.6.235.67" port: 42213 } } peers { permanent_uuid: "931a82ffd0ef48a48003604170d99a22" member_type: VOTER last_known_addr { host: "127.6.235.66" port: 40407 } }
I20260928 02:10:44.350512 8717 raft_consensus.cc:374] T 8981328c9d994a0ea7cafbfebe3132b2 P 1296711f54774aa2880bc8d1d2ec7b70 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:10:44.350701 8717 raft_consensus.cc:729] T 8981328c9d994a0ea7cafbfebe3132b2 P 1296711f54774aa2880bc8d1d2ec7b70 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1296711f54774aa2880bc8d1d2ec7b70, State: Initialized, Role: FOLLOWER
I20260928 02:10:44.351569 8717 consensus_queue.cc:260] T 8981328c9d994a0ea7cafbfebe3132b2 P 1296711f54774aa2880bc8d1d2ec7b70 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1296711f54774aa2880bc8d1d2ec7b70" member_type: VOTER last_known_addr { host: "127.6.235.65" port: 34341 } } peers { permanent_uuid: "e84d3c7d475a4ad9a1d40d8938b60cf7" member_type: VOTER last_known_addr { host: "127.6.235.67" port: 42213 } } peers { permanent_uuid: "931a82ffd0ef48a48003604170d99a22" member_type: VOTER last_known_addr { host: "127.6.235.66" port: 40407 } }
I20260928 02:10:44.353902 8717 ts_tablet_manager.cc:1428] T 8981328c9d994a0ea7cafbfebe3132b2 P 1296711f54774aa2880bc8d1d2ec7b70: Time spent starting tablet: real 0.015s user 0.007s sys 0.009s
I20260928 02:10:44.355866 8718 raft_consensus.cc:348] T 8981328c9d994a0ea7cafbfebe3132b2 P e84d3c7d475a4ad9a1d40d8938b60cf7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1296711f54774aa2880bc8d1d2ec7b70" member_type: VOTER last_known_addr { host: "127.6.235.65" port: 34341 } } peers { permanent_uuid: "e84d3c7d475a4ad9a1d40d8938b60cf7" member_type: VOTER last_known_addr { host: "127.6.235.67" port: 42213 } } peers { permanent_uuid: "931a82ffd0ef48a48003604170d99a22" member_type: VOTER last_known_addr { host: "127.6.235.66" port: 40407 } }
I20260928 02:10:44.356317 8718 raft_consensus.cc:374] T 8981328c9d994a0ea7cafbfebe3132b2 P e84d3c7d475a4ad9a1d40d8938b60cf7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:10:44.356427 8718 raft_consensus.cc:729] T 8981328c9d994a0ea7cafbfebe3132b2 P e84d3c7d475a4ad9a1d40d8938b60cf7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e84d3c7d475a4ad9a1d40d8938b60cf7, State: Initialized, Role: FOLLOWER
I20260928 02:10:44.357199 8558 heartbeater.cc:503] Master 127.6.235.126:37431 was elected leader, sending a full tablet report...
I20260928 02:10:44.357170 8718 consensus_queue.cc:260] T 8981328c9d994a0ea7cafbfebe3132b2 P e84d3c7d475a4ad9a1d40d8938b60cf7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1296711f54774aa2880bc8d1d2ec7b70" member_type: VOTER last_known_addr { host: "127.6.235.65" port: 34341 } } peers { permanent_uuid: "e84d3c7d475a4ad9a1d40d8938b60cf7" member_type: VOTER last_known_addr { host: "127.6.235.67" port: 42213 } } peers { permanent_uuid: "931a82ffd0ef48a48003604170d99a22" member_type: VOTER last_known_addr { host: "127.6.235.66" port: 40407 } }
I20260928 02:10:44.357893 8716 ts_tablet_manager.cc:1428] T 8981328c9d994a0ea7cafbfebe3132b2 P 931a82ffd0ef48a48003604170d99a22: Time spent starting tablet: real 0.023s user 0.013s sys 0.009s
I20260928 02:10:44.359273 8697 heartbeater.cc:503] Master 127.6.235.126:37431 was elected leader, sending a full tablet report...
I20260928 02:10:44.359951 8718 ts_tablet_manager.cc:1428] T 8981328c9d994a0ea7cafbfebe3132b2 P e84d3c7d475a4ad9a1d40d8938b60cf7: Time spent starting tablet: real 0.013s user 0.014s sys 0.000s
W20260928 02:10:44.386417 8420 tablet.cc:2369] T 8981328c9d994a0ea7cafbfebe3132b2 P 1296711f54774aa2880bc8d1d2ec7b70: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 02:10:44.390667 8722 raft_consensus.cc:482] T 8981328c9d994a0ea7cafbfebe3132b2 P 931a82ffd0ef48a48003604170d99a22 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 02:10:44.391021 8722 raft_consensus.cc:504] T 8981328c9d994a0ea7cafbfebe3132b2 P 931a82ffd0ef48a48003604170d99a22 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1296711f54774aa2880bc8d1d2ec7b70" member_type: VOTER last_known_addr { host: "127.6.235.65" port: 34341 } } peers { permanent_uuid: "e84d3c7d475a4ad9a1d40d8938b60cf7" member_type: VOTER last_known_addr { host: "127.6.235.67" port: 42213 } } peers { permanent_uuid: "931a82ffd0ef48a48003604170d99a22" member_type: VOTER last_known_addr { host: "127.6.235.66" port: 40407 } }
I20260928 02:10:44.393031 8722 leader_election.cc:288] T 8981328c9d994a0ea7cafbfebe3132b2 P 931a82ffd0ef48a48003604170d99a22 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1296711f54774aa2880bc8d1d2ec7b70 (127.6.235.65:34341), e84d3c7d475a4ad9a1d40d8938b60cf7 (127.6.235.67:42213)
I20260928 02:10:44.400041 8713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:10:44.393524 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.235.66:39219 (local address 127.6.235.67:42213)
0928 02:10:44.393731 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0928 02:10:44.393739 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 02:10:44.394241 (+ 502us) server_negotiation.cc:407] Connection header received
0928 02:10:44.394475 (+ 234us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:10:44.394479 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:10:44.394544 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:10:44.394638 (+ 94us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:10:44.395183 (+ 545us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:10:44.395870 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:10:44.398181 (+ 2311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:10:44.399083 (+ 902us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:10:44.399284 (+ 201us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:10:44.399363 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0928 02:10:44.399651 (+ 288us) server_negotiation.cc:299] Negotiation successful
0928 02:10:44.399772 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260928 02:10:44.400202 8712 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:10:44.393360 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.235.66:46003 (local address 127.6.235.65:34341)
0928 02:10:44.393618 (+ 258us) server_negotiation.cc:206] Beginning negotiation
0928 02:10:44.393624 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 02:10:44.394509 (+ 885us) server_negotiation.cc:407] Connection header received
0928 02:10:44.394713 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:10:44.394719 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:10:44.394785 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:10:44.394890 (+ 105us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:10:44.395441 (+ 551us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:10:44.396129 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:10:44.398485 (+ 2356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:10:44.399262 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:10:44.399447 (+ 185us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:10:44.399522 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0928 02:10:44.399807 (+ 285us) server_negotiation.cc:299] Negotiation successful
0928 02:10:44.399948 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":109}
I20260928 02:10:44.400354 8726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:10:44.393594 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.235.65:34341 (local address 127.6.235.66:46003)
0928 02:10:44.394391 (+ 797us) negotiation.cc:107] Waiting for socket to connect
0928 02:10:44.394406 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 02:10:44.394572 (+ 166us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:10:44.394903 (+ 331us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:10:44.394907 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:10:44.394933 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:10:44.395295 (+ 362us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:10:44.395303 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:10:44.396267 (+ 964us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:10:44.396276 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:10:44.398271 (+ 1995us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:10:44.398287 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:10:44.399489 (+ 1202us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:10:44.399495 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:10:44.399655 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:10:44.399678 (+ 23us) client_negotiation.cc:769] Sending connection context
0928 02:10:44.399783 (+ 105us) client_negotiation.cc:240] Negotiation successful
0928 02:10:44.399921 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":664,"mutex_wait_us":44,"thread_start_us":485,"threads_started":1}
I20260928 02:10:44.400379 8725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:10:44.393335 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.235.67:42213 (local address 127.6.235.66:39219)
0928 02:10:44.394116 (+ 781us) negotiation.cc:107] Waiting for socket to connect
0928 02:10:44.394134 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0928 02:10:44.394326 (+ 192us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:10:44.394666 (+ 340us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:10:44.394671 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:10:44.394694 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:10:44.395040 (+ 346us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:10:44.395051 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:10:44.396009 (+ 958us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:10:44.396015 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:10:44.398037 (+ 2022us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:10:44.398049 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:10:44.399300 (+ 1251us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:10:44.399306 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:10:44.399486 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:10:44.399518 (+ 32us) client_negotiation.cc:769] Sending connection context
0928 02:10:44.399627 (+ 109us) client_negotiation.cc:240] Negotiation successful
0928 02:10:44.399761 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":614,"mutex_wait_us":104,"thread_start_us":401,"threads_started":1}
I20260928 02:10:44.401413 8369 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8981328c9d994a0ea7cafbfebe3132b2" candidate_uuid: "931a82ffd0ef48a48003604170d99a22" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1296711f54774aa2880bc8d1d2ec7b70" is_pre_election: true
I20260928 02:10:44.401417 8647 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8981328c9d994a0ea7cafbfebe3132b2" candidate_uuid: "931a82ffd0ef48a48003604170d99a22" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e84d3c7d475a4ad9a1d40d8938b60cf7" is_pre_election: true
I20260928 02:10:44.401934 8369 raft_consensus.cc:2454] T 8981328c9d994a0ea7cafbfebe3132b2 P 1296711f54774aa2880bc8d1d2ec7b70 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 931a82ffd0ef48a48003604170d99a22 in term 0.
I20260928 02:10:44.401934 8647 raft_consensus.cc:2454] T 8981328c9d994a0ea7cafbfebe3132b2 P e84d3c7d475a4ad9a1d40d8938b60cf7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 931a82ffd0ef48a48003604170d99a22 in term 0.
I20260928 02:10:44.402595 8441 leader_election.cc:302] T 8981328c9d994a0ea7cafbfebe3132b2 P 931a82ffd0ef48a48003604170d99a22 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 1296711f54774aa2880bc8d1d2ec7b70, 931a82ffd0ef48a48003604170d99a22, e84d3c7d475a4ad9a1d40d8938b60cf7; no voters:
I20260928 02:10:44.402995 8722 raft_consensus.cc:2781] T 8981328c9d994a0ea7cafbfebe3132b2 P 931a82ffd0ef48a48003604170d99a22 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 02:10:44.403146 8722 raft_consensus.cc:482] T 8981328c9d994a0ea7cafbfebe3132b2 P 931a82ffd0ef48a48003604170d99a22 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 02:10:44.403254 8722 raft_consensus.cc:3037] T 8981328c9d994a0ea7cafbfebe3132b2 P 931a82ffd0ef48a48003604170d99a22 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:10:44.405544 8722 raft_consensus.cc:504] T 8981328c9d994a0ea7cafbfebe3132b2 P 931a82ffd0ef48a48003604170d99a22 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1296711f54774aa2880bc8d1d2ec7b70" member_type: VOTER last_known_addr { host: "127.6.235.65" port: 34341 } } peers { permanent_uuid: "e84d3c7d475a4ad9a1d40d8938b60cf7" member_type: VOTER last_known_addr { host: "127.6.235.67" port: 42213 } } peers { permanent_uuid: "931a82ffd0ef48a48003604170d99a22" member_type: VOTER last_known_addr { host: "127.6.235.66" port: 40407 } }
I20260928 02:10:44.406188 8722 leader_election.cc:288] T 8981328c9d994a0ea7cafbfebe3132b2 P 931a82ffd0ef48a48003604170d99a22 [CANDIDATE]: Term 1 election: Requested vote from peers 1296711f54774aa2880bc8d1d2ec7b70 (127.6.235.65:34341), e84d3c7d475a4ad9a1d40d8938b60cf7 (127.6.235.67:42213)
I20260928 02:10:44.406615 8369 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8981328c9d994a0ea7cafbfebe3132b2" candidate_uuid: "931a82ffd0ef48a48003604170d99a22" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1296711f54774aa2880bc8d1d2ec7b70"
I20260928 02:10:44.406733 8647 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8981328c9d994a0ea7cafbfebe3132b2" candidate_uuid: "931a82ffd0ef48a48003604170d99a22" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e84d3c7d475a4ad9a1d40d8938b60cf7"
I20260928 02:10:44.406844 8369 raft_consensus.cc:3037] T 8981328c9d994a0ea7cafbfebe3132b2 P 1296711f54774aa2880bc8d1d2ec7b70 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:10:44.406924 8647 raft_consensus.cc:3037] T 8981328c9d994a0ea7cafbfebe3132b2 P e84d3c7d475a4ad9a1d40d8938b60cf7 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:10:44.410174 8369 raft_consensus.cc:2454] T 8981328c9d994a0ea7cafbfebe3132b2 P 1296711f54774aa2880bc8d1d2ec7b70 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 931a82ffd0ef48a48003604170d99a22 in term 1.
I20260928 02:10:44.410173 8647 raft_consensus.cc:2454] T 8981328c9d994a0ea7cafbfebe3132b2 P e84d3c7d475a4ad9a1d40d8938b60cf7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 931a82ffd0ef48a48003604170d99a22 in term 1.
I20260928 02:10:44.410683 8440 leader_election.cc:302] T 8981328c9d994a0ea7cafbfebe3132b2 P 931a82ffd0ef48a48003604170d99a22 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1296711f54774aa2880bc8d1d2ec7b70, 931a82ffd0ef48a48003604170d99a22; no voters:
I20260928 02:10:44.411015 8722 raft_consensus.cc:2781] T 8981328c9d994a0ea7cafbfebe3132b2 P 931a82ffd0ef48a48003604170d99a22 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:10:44.411558 8722 raft_consensus.cc:686] T 8981328c9d994a0ea7cafbfebe3132b2 P 931a82ffd0ef48a48003604170d99a22 [term 1 LEADER]: Becoming Leader. State: Replica: 931a82ffd0ef48a48003604170d99a22, State: Running, Role: LEADER
I20260928 02:10:44.412053 8722 consensus_queue.cc:237] T 8981328c9d994a0ea7cafbfebe3132b2 P 931a82ffd0ef48a48003604170d99a22 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1296711f54774aa2880bc8d1d2ec7b70" member_type: VOTER last_known_addr { host: "127.6.235.65" port: 34341 } } peers { permanent_uuid: "e84d3c7d475a4ad9a1d40d8938b60cf7" member_type: VOTER last_known_addr { host: "127.6.235.67" port: 42213 } } peers { permanent_uuid: "931a82ffd0ef48a48003604170d99a22" member_type: VOTER last_known_addr { host: "127.6.235.66" port: 40407 } }
I20260928 02:10:44.417265 8211 catalog_manager.cc:5495] T 8981328c9d994a0ea7cafbfebe3132b2 P 931a82ffd0ef48a48003604170d99a22 reported cstate change: term changed from 0 to 1, leader changed from <none> to 931a82ffd0ef48a48003604170d99a22 (127.6.235.66). New cstate: current_term: 1 leader_uuid: "931a82ffd0ef48a48003604170d99a22" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1296711f54774aa2880bc8d1d2ec7b70" member_type: VOTER last_known_addr { host: "127.6.235.65" port: 34341 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e84d3c7d475a4ad9a1d40d8938b60cf7" member_type: VOTER last_known_addr { host: "127.6.235.67" port: 42213 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "931a82ffd0ef48a48003604170d99a22" member_type: VOTER last_known_addr { host: "127.6.235.66" port: 40407 } health_report { overall_health: HEALTHY } } }
W20260928 02:10:44.420567 8559 tablet.cc:2369] T 8981328c9d994a0ea7cafbfebe3132b2 P 931a82ffd0ef48a48003604170d99a22: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 02:10:44.434405 8698 tablet.cc:2369] T 8981328c9d994a0ea7cafbfebe3132b2 P e84d3c7d475a4ad9a1d40d8938b60cf7: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 02:10:44.441202 8715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:10:44.434893 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34856 (local address 127.6.235.66:40407)
0928 02:10:44.435099 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0928 02:10:44.435106 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 02:10:44.435412 (+ 306us) server_negotiation.cc:407] Connection header received
0928 02:10:44.435793 (+ 381us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:10:44.435797 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:10:44.435886 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:10:44.436044 (+ 158us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 02:10:44.436422 (+ 378us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:10:44.437268 (+ 846us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:10:44.438832 (+ 1564us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:10:44.439339 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:10:44.439488 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:10:44.439973 (+ 485us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 02:10:44.440509 (+ 536us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 02:10:44.440671 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0928 02:10:44.440813 (+ 142us) server_negotiation.cc:299] Negotiation successful
0928 02:10:44.440930 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260928 02:10:44.450289 8647 raft_consensus.cc:1261] T 8981328c9d994a0ea7cafbfebe3132b2 P e84d3c7d475a4ad9a1d40d8938b60cf7 [term 1 FOLLOWER]: Refusing update from remote peer 931a82ffd0ef48a48003604170d99a22: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 02:10:44.450294 8369 raft_consensus.cc:1261] T 8981328c9d994a0ea7cafbfebe3132b2 P 1296711f54774aa2880bc8d1d2ec7b70 [term 1 FOLLOWER]: Refusing update from remote peer 931a82ffd0ef48a48003604170d99a22: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 02:10:44.451283 8727 consensus_queue.cc:1035] T 8981328c9d994a0ea7cafbfebe3132b2 P 931a82ffd0ef48a48003604170d99a22 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e84d3c7d475a4ad9a1d40d8938b60cf7" member_type: VOTER last_known_addr { host: "127.6.235.67" port: 42213 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 02:10:44.451690 8722 consensus_queue.cc:1035] T 8981328c9d994a0ea7cafbfebe3132b2 P 931a82ffd0ef48a48003604170d99a22 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1296711f54774aa2880bc8d1d2ec7b70" member_type: VOTER last_known_addr { host: "127.6.235.65" port: 34341 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 02:10:44.463419 8730 mvcc.cc:205] Tried to move back new op lower bound from 7334139676456546304 to 7334139676316250112. Current Snapshot: MvccSnapshot[applied={T|T < 7334139676456546304}]
I20260928 02:10:44.483929 8212 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:46612:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 02:10:44.484165 8212 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:46612:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 02:10:44.488075 8212 catalog_manager.cc:5782] T 00000000000000000000000000000000 P ae6ca031c9744a50b87bd00e3e3df444: Sending DeleteTablet for 3 replicas of tablet 8981328c9d994a0ea7cafbfebe3132b2
I20260928 02:10:44.489440 8627 tablet_service.cc:1514] Processing DeleteTablet for tablet 8981328c9d994a0ea7cafbfebe3132b2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:10:44 UTC) from {username='kudu', principal='kudu/127.6.235.126@KRBTEST.COM'} at 127.0.0.1:45692
I20260928 02:10:44.489426 8349 tablet_service.cc:1514] Processing DeleteTablet for tablet 8981328c9d994a0ea7cafbfebe3132b2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:10:44 UTC) from {username='kudu', principal='kudu/127.6.235.126@KRBTEST.COM'} at 127.0.0.1:47860
I20260928 02:10:44.489500 8488 tablet_service.cc:1514] Processing DeleteTablet for tablet 8981328c9d994a0ea7cafbfebe3132b2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:10:44 UTC) from {username='kudu', principal='kudu/127.6.235.126@KRBTEST.COM'} at 127.0.0.1:34854
I20260928 02:10:44.492301 8212 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:46612:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 02:10:44.493279 8212 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 02:10:44.505725 8487 tablet_service.cc:1467] Processing CreateTablet for tablet a54b2154ec784eaa84129bbf5d1b904e (DEFAULT_TABLE table=test-table [id=f60e0dc0b21a41009f53d4f6e7dedbf1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:10:44.506108 8626 tablet_service.cc:1467] Processing CreateTablet for tablet a54b2154ec784eaa84129bbf5d1b904e (DEFAULT_TABLE table=test-table [id=f60e0dc0b21a41009f53d4f6e7dedbf1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:10:44.506392 8487 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a54b2154ec784eaa84129bbf5d1b904e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:10:44.506841 8626 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a54b2154ec784eaa84129bbf5d1b904e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:10:44.507712 8740 tablet_replica.cc:330] T 8981328c9d994a0ea7cafbfebe3132b2 P 1296711f54774aa2880bc8d1d2ec7b70: stopping tablet replica
I20260928 02:10:44.508213 8740 raft_consensus.cc:2229] T 8981328c9d994a0ea7cafbfebe3132b2 P 1296711f54774aa2880bc8d1d2ec7b70 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 02:10:44.508767 8740 raft_consensus.cc:2258] T 8981328c9d994a0ea7cafbfebe3132b2 P 1296711f54774aa2880bc8d1d2ec7b70 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 02:10:44.510119 8348 tablet_service.cc:1467] Processing CreateTablet for tablet a54b2154ec784eaa84129bbf5d1b904e (DEFAULT_TABLE table=test-table [id=f60e0dc0b21a41009f53d4f6e7dedbf1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:10:44.510578 8348 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a54b2154ec784eaa84129bbf5d1b904e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:10:44.513631 8716 tablet_bootstrap.cc:492] T a54b2154ec784eaa84129bbf5d1b904e P 931a82ffd0ef48a48003604170d99a22: Bootstrap starting.
I20260928 02:10:44.515869 8716 tablet_bootstrap.cc:654] T a54b2154ec784eaa84129bbf5d1b904e P 931a82ffd0ef48a48003604170d99a22: Neither blocks nor log segments found. Creating new log.
I20260928 02:10:44.516376 8718 tablet_bootstrap.cc:492] T a54b2154ec784eaa84129bbf5d1b904e P e84d3c7d475a4ad9a1d40d8938b60cf7: Bootstrap starting.
I20260928 02:10:44.517354 8742 tablet_replica.cc:330] T 8981328c9d994a0ea7cafbfebe3132b2 P 931a82ffd0ef48a48003604170d99a22: stopping tablet replica
I20260928 02:10:44.517856 8742 raft_consensus.cc:2229] T 8981328c9d994a0ea7cafbfebe3132b2 P 931a82ffd0ef48a48003604170d99a22 [term 1 LEADER]: Raft consensus shutting down.
I20260928 02:10:44.518200 8718 tablet_bootstrap.cc:654] T a54b2154ec784eaa84129bbf5d1b904e P e84d3c7d475a4ad9a1d40d8938b60cf7: Neither blocks nor log segments found. Creating new log.
I20260928 02:10:44.518646 8742 raft_consensus.cc:2258] T 8981328c9d994a0ea7cafbfebe3132b2 P 931a82ffd0ef48a48003604170d99a22 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 02:10:44.522149 8717 tablet_bootstrap.cc:492] T a54b2154ec784eaa84129bbf5d1b904e P 1296711f54774aa2880bc8d1d2ec7b70: Bootstrap starting.
I20260928 02:10:44.524650 8717 tablet_bootstrap.cc:654] T a54b2154ec784eaa84129bbf5d1b904e P 1296711f54774aa2880bc8d1d2ec7b70: Neither blocks nor log segments found. Creating new log.
I20260928 02:10:44.509578 8741 tablet_replica.cc:330] T 8981328c9d994a0ea7cafbfebe3132b2 P e84d3c7d475a4ad9a1d40d8938b60cf7: stopping tablet replica
I20260928 02:10:44.525938 8741 raft_consensus.cc:2229] T 8981328c9d994a0ea7cafbfebe3132b2 P e84d3c7d475a4ad9a1d40d8938b60cf7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 02:10:44.526680 8741 raft_consensus.cc:2258] T 8981328c9d994a0ea7cafbfebe3132b2 P e84d3c7d475a4ad9a1d40d8938b60cf7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 02:10:44.528172 8716 tablet_bootstrap.cc:492] T a54b2154ec784eaa84129bbf5d1b904e P 931a82ffd0ef48a48003604170d99a22: No bootstrap required, opened a new log
I20260928 02:10:44.528422 8716 ts_tablet_manager.cc:1397] T a54b2154ec784eaa84129bbf5d1b904e P 931a82ffd0ef48a48003604170d99a22: Time spent bootstrapping tablet: real 0.015s user 0.006s sys 0.003s
I20260928 02:10:44.529474 8716 raft_consensus.cc:348] T a54b2154ec784eaa84129bbf5d1b904e P 931a82ffd0ef48a48003604170d99a22 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1296711f54774aa2880bc8d1d2ec7b70" member_type: VOTER last_known_addr { host: "127.6.235.65" port: 34341 } } peers { permanent_uuid: "931a82ffd0ef48a48003604170d99a22" member_type: VOTER last_known_addr { host: "127.6.235.66" port: 40407 } } peers { permanent_uuid: "e84d3c7d475a4ad9a1d40d8938b60cf7" member_type: VOTER last_known_addr { host: "127.6.235.67" port: 42213 } }
I20260928 02:10:44.529811 8716 raft_consensus.cc:374] T a54b2154ec784eaa84129bbf5d1b904e P 931a82ffd0ef48a48003604170d99a22 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:10:44.529906 8718 tablet_bootstrap.cc:492] T a54b2154ec784eaa84129bbf5d1b904e P e84d3c7d475a4ad9a1d40d8938b60cf7: No bootstrap required, opened a new log
I20260928 02:10:44.529949 8716 raft_consensus.cc:729] T a54b2154ec784eaa84129bbf5d1b904e P 931a82ffd0ef48a48003604170d99a22 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 931a82ffd0ef48a48003604170d99a22, State: Initialized, Role: FOLLOWER
I20260928 02:10:44.530130 8718 ts_tablet_manager.cc:1397] T a54b2154ec784eaa84129bbf5d1b904e P e84d3c7d475a4ad9a1d40d8938b60cf7: Time spent bootstrapping tablet: real 0.014s user 0.005s sys 0.008s
I20260928 02:10:44.530264 8716 consensus_queue.cc:260] T a54b2154ec784eaa84129bbf5d1b904e P 931a82ffd0ef48a48003604170d99a22 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1296711f54774aa2880bc8d1d2ec7b70" member_type: VOTER last_known_addr { host: "127.6.235.65" port: 34341 } } peers { permanent_uuid: "931a82ffd0ef48a48003604170d99a22" member_type: VOTER last_known_addr { host: "127.6.235.66" port: 40407 } } peers { permanent_uuid: "e84d3c7d475a4ad9a1d40d8938b60cf7" member_type: VOTER last_known_addr { host: "127.6.235.67" port: 42213 } }
I20260928 02:10:44.531042 8716 ts_tablet_manager.cc:1428] T a54b2154ec784eaa84129bbf5d1b904e P 931a82ffd0ef48a48003604170d99a22: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260928 02:10:44.531035 8718 raft_consensus.cc:348] T a54b2154ec784eaa84129bbf5d1b904e P e84d3c7d475a4ad9a1d40d8938b60cf7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1296711f54774aa2880bc8d1d2ec7b70" member_type: VOTER last_known_addr { host: "127.6.235.65" port: 34341 } } peers { permanent_uuid: "931a82ffd0ef48a48003604170d99a22" member_type: VOTER last_known_addr { host: "127.6.235.66" port: 40407 } } peers { permanent_uuid: "e84d3c7d475a4ad9a1d40d8938b60cf7" member_type: VOTER last_known_addr { host: "127.6.235.67" port: 42213 } }
I20260928 02:10:44.531296 8718 raft_consensus.cc:374] T a54b2154ec784eaa84129bbf5d1b904e P e84d3c7d475a4ad9a1d40d8938b60cf7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:10:44.531412 8718 raft_consensus.cc:729] T a54b2154ec784eaa84129bbf5d1b904e P e84d3c7d475a4ad9a1d40d8938b60cf7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e84d3c7d475a4ad9a1d40d8938b60cf7, State: Initialized, Role: FOLLOWER
I20260928 02:10:44.531737 8718 consensus_queue.cc:260] T a54b2154ec784eaa84129bbf5d1b904e P e84d3c7d475a4ad9a1d40d8938b60cf7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1296711f54774aa2880bc8d1d2ec7b70" member_type: VOTER last_known_addr { host: "127.6.235.65" port: 34341 } } peers { permanent_uuid: "931a82ffd0ef48a48003604170d99a22" member_type: VOTER last_known_addr { host: "127.6.235.66" port: 40407 } } peers { permanent_uuid: "e84d3c7d475a4ad9a1d40d8938b60cf7" member_type: VOTER last_known_addr { host: "127.6.235.67" port: 42213 } }
I20260928 02:10:44.533409 8718 ts_tablet_manager.cc:1428] T a54b2154ec784eaa84129bbf5d1b904e P e84d3c7d475a4ad9a1d40d8938b60cf7: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20260928 02:10:44.533543 8741 ts_tablet_manager.cc:1905] T 8981328c9d994a0ea7cafbfebe3132b2 P e84d3c7d475a4ad9a1d40d8938b60cf7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 02:10:44.533390 8740 ts_tablet_manager.cc:1905] T 8981328c9d994a0ea7cafbfebe3132b2 P 1296711f54774aa2880bc8d1d2ec7b70: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 02:10:44.541612 8741 ts_tablet_manager.cc:1918] T 8981328c9d994a0ea7cafbfebe3132b2 P e84d3c7d475a4ad9a1d40d8938b60cf7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 02:10:44.541842 8741 log.cc:1196] T 8981328c9d994a0ea7cafbfebe3132b2 P e84d3c7d475a4ad9a1d40d8938b60cf7: Deleting WAL directory at /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790561432251112-7085-0/minicluster-data/ts-2/wal/wals/8981328c9d994a0ea7cafbfebe3132b2
I20260928 02:10:44.542498 8741 ts_tablet_manager.cc:1939] T 8981328c9d994a0ea7cafbfebe3132b2 P e84d3c7d475a4ad9a1d40d8938b60cf7: Deleting consensus metadata
I20260928 02:10:44.543716 8717 tablet_bootstrap.cc:492] T a54b2154ec784eaa84129bbf5d1b904e P 1296711f54774aa2880bc8d1d2ec7b70: No bootstrap required, opened a new log
I20260928 02:10:44.543934 8717 ts_tablet_manager.cc:1397] T a54b2154ec784eaa84129bbf5d1b904e P 1296711f54774aa2880bc8d1d2ec7b70: Time spent bootstrapping tablet: real 0.022s user 0.006s sys 0.000s
I20260928 02:10:44.544724 8198 catalog_manager.cc:4841] TS e84d3c7d475a4ad9a1d40d8938b60cf7 (127.6.235.67:42213): tablet 8981328c9d994a0ea7cafbfebe3132b2 (table test-table [id=e40ea3ba5e6949488e406fcb75d0ab59]) successfully deleted
I20260928 02:10:44.544859 8717 raft_consensus.cc:348] T a54b2154ec784eaa84129bbf5d1b904e P 1296711f54774aa2880bc8d1d2ec7b70 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1296711f54774aa2880bc8d1d2ec7b70" member_type: VOTER last_known_addr { host: "127.6.235.65" port: 34341 } } peers { permanent_uuid: "931a82ffd0ef48a48003604170d99a22" member_type: VOTER last_known_addr { host: "127.6.235.66" port: 40407 } } peers { permanent_uuid: "e84d3c7d475a4ad9a1d40d8938b60cf7" member_type: VOTER last_known_addr { host: "127.6.235.67" port: 42213 } }
I20260928 02:10:44.545164 8717 raft_consensus.cc:374] T a54b2154ec784eaa84129bbf5d1b904e P 1296711f54774aa2880bc8d1d2ec7b70 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:10:44.545290 8717 raft_consensus.cc:729] T a54b2154ec784eaa84129bbf5d1b904e P 1296711f54774aa2880bc8d1d2ec7b70 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1296711f54774aa2880bc8d1d2ec7b70, State: Initialized, Role: FOLLOWER
I20260928 02:10:44.545468 8742 ts_tablet_manager.cc:1905] T 8981328c9d994a0ea7cafbfebe3132b2 P 931a82ffd0ef48a48003604170d99a22: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 02:10:44.545694 8717 consensus_queue.cc:260] T a54b2154ec784eaa84129bbf5d1b904e P 1296711f54774aa2880bc8d1d2ec7b70 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1296711f54774aa2880bc8d1d2ec7b70" member_type: VOTER last_known_addr { host: "127.6.235.65" port: 34341 } } peers { permanent_uuid: "931a82ffd0ef48a48003604170d99a22" member_type: VOTER last_known_addr { host: "127.6.235.66" port: 40407 } } peers { permanent_uuid: "e84d3c7d475a4ad9a1d40d8938b60cf7" member_type: VOTER last_known_addr { host: "127.6.235.67" port: 42213 } }
I20260928 02:10:44.548353 8740 ts_tablet_manager.cc:1918] T 8981328c9d994a0ea7cafbfebe3132b2 P 1296711f54774aa2880bc8d1d2ec7b70: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 02:10:44.548513 8717 ts_tablet_manager.cc:1428] T a54b2154ec784eaa84129bbf5d1b904e P 1296711f54774aa2880bc8d1d2ec7b70: Time spent starting tablet: real 0.004s user 0.000s sys 0.003s
I20260928 02:10:44.548656 8740 log.cc:1196] T 8981328c9d994a0ea7cafbfebe3132b2 P 1296711f54774aa2880bc8d1d2ec7b70: Deleting WAL directory at /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790561432251112-7085-0/minicluster-data/ts-0/wal/wals/8981328c9d994a0ea7cafbfebe3132b2
I20260928 02:10:44.549295 8740 ts_tablet_manager.cc:1939] T 8981328c9d994a0ea7cafbfebe3132b2 P 1296711f54774aa2880bc8d1d2ec7b70: Deleting consensus metadata
I20260928 02:10:44.551154 8197 catalog_manager.cc:4841] TS 1296711f54774aa2880bc8d1d2ec7b70 (127.6.235.65:34341): tablet 8981328c9d994a0ea7cafbfebe3132b2 (table test-table [id=e40ea3ba5e6949488e406fcb75d0ab59]) successfully deleted
I20260928 02:10:44.554884 8742 ts_tablet_manager.cc:1918] T 8981328c9d994a0ea7cafbfebe3132b2 P 931a82ffd0ef48a48003604170d99a22: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 02:10:44.555089 8742 log.cc:1196] T 8981328c9d994a0ea7cafbfebe3132b2 P 931a82ffd0ef48a48003604170d99a22: Deleting WAL directory at /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790561432251112-7085-0/minicluster-data/ts-1/wal/wals/8981328c9d994a0ea7cafbfebe3132b2
I20260928 02:10:44.555678 8742 ts_tablet_manager.cc:1939] T 8981328c9d994a0ea7cafbfebe3132b2 P 931a82ffd0ef48a48003604170d99a22: Deleting consensus metadata
I20260928 02:10:44.557260 8198 catalog_manager.cc:4841] TS 931a82ffd0ef48a48003604170d99a22 (127.6.235.66:40407): tablet 8981328c9d994a0ea7cafbfebe3132b2 (table test-table [id=e40ea3ba5e6949488e406fcb75d0ab59]) successfully deleted
I20260928 02:10:44.670270 8724 raft_consensus.cc:482] T a54b2154ec784eaa84129bbf5d1b904e P e84d3c7d475a4ad9a1d40d8938b60cf7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 02:10:44.670526 8724 raft_consensus.cc:504] T a54b2154ec784eaa84129bbf5d1b904e P e84d3c7d475a4ad9a1d40d8938b60cf7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1296711f54774aa2880bc8d1d2ec7b70" member_type: VOTER last_known_addr { host: "127.6.235.65" port: 34341 } } peers { permanent_uuid: "931a82ffd0ef48a48003604170d99a22" member_type: VOTER last_known_addr { host: "127.6.235.66" port: 40407 } } peers { permanent_uuid: "e84d3c7d475a4ad9a1d40d8938b60cf7" member_type: VOTER last_known_addr { host: "127.6.235.67" port: 42213 } }
I20260928 02:10:44.672394 8724 leader_election.cc:288] T a54b2154ec784eaa84129bbf5d1b904e P e84d3c7d475a4ad9a1d40d8938b60cf7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1296711f54774aa2880bc8d1d2ec7b70 (127.6.235.65:34341), 931a82ffd0ef48a48003604170d99a22 (127.6.235.66:40407)
I20260928 02:10:44.691120 8712 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:10:44.673199 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.235.67:37859 (local address 127.6.235.65:34341)
0928 02:10:44.673449 (+ 250us) server_negotiation.cc:206] Beginning negotiation
0928 02:10:44.673458 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 02:10:44.684051 (+ 10593us) server_negotiation.cc:407] Connection header received
0928 02:10:44.684246 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:10:44.684252 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:10:44.684345 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:10:44.684434 (+ 89us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:10:44.685001 (+ 567us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:10:44.686105 (+ 1104us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:10:44.688644 (+ 2539us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:10:44.689907 (+ 1263us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:10:44.690042 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:10:44.690126 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0928 02:10:44.690685 (+ 559us) server_negotiation.cc:299] Negotiation successful
0928 02:10:44.690827 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
I20260928 02:10:44.691901 8746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:10:44.672604 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.235.65:34341 (local address 127.6.235.67:37859)
0928 02:10:44.683917 (+ 11313us) negotiation.cc:107] Waiting for socket to connect
0928 02:10:44.683940 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0928 02:10:44.684116 (+ 176us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:10:44.684451 (+ 335us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:10:44.684456 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:10:44.684480 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:10:44.684818 (+ 338us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:10:44.684830 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:10:44.686252 (+ 1422us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:10:44.686259 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:10:44.688447 (+ 2188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:10:44.688460 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:10:44.690303 (+ 1843us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:10:44.690309 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:10:44.690495 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:10:44.690531 (+ 36us) client_negotiation.cc:769] Sending connection context
0928 02:10:44.691390 (+ 859us) client_negotiation.cc:240] Negotiation successful
0928 02:10:44.691547 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11160,"thread_start_us":11602,"threads_started":1}
I20260928 02:10:44.692730 8369 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a54b2154ec784eaa84129bbf5d1b904e" candidate_uuid: "e84d3c7d475a4ad9a1d40d8938b60cf7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1296711f54774aa2880bc8d1d2ec7b70" is_pre_election: true
I20260928 02:10:44.693027 8369 raft_consensus.cc:2454] T a54b2154ec784eaa84129bbf5d1b904e P 1296711f54774aa2880bc8d1d2ec7b70 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e84d3c7d475a4ad9a1d40d8938b60cf7 in term 0.
I20260928 02:10:44.693485 8715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:10:44.673644 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.235.67:57335 (local address 127.6.235.66:40407)
0928 02:10:44.673858 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0928 02:10:44.673867 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 02:10:44.685592 (+ 11725us) server_negotiation.cc:407] Connection header received
0928 02:10:44.685768 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:10:44.685774 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:10:44.685908 (+ 134us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:10:44.685970 (+ 62us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:10:44.686855 (+ 885us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:10:44.687980 (+ 1125us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:10:44.690389 (+ 2409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:10:44.691669 (+ 1280us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:10:44.691792 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:10:44.691858 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0928 02:10:44.693009 (+ 1151us) server_negotiation.cc:299] Negotiation successful
0928 02:10:44.693174 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260928 02:10:44.693558 8579 leader_election.cc:302] T a54b2154ec784eaa84129bbf5d1b904e P e84d3c7d475a4ad9a1d40d8938b60cf7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1296711f54774aa2880bc8d1d2ec7b70, e84d3c7d475a4ad9a1d40d8938b60cf7; no voters:
I20260928 02:10:44.694028 8724 raft_consensus.cc:2781] T a54b2154ec784eaa84129bbf5d1b904e P e84d3c7d475a4ad9a1d40d8938b60cf7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 02:10:44.694149 8724 raft_consensus.cc:482] T a54b2154ec784eaa84129bbf5d1b904e P e84d3c7d475a4ad9a1d40d8938b60cf7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 02:10:44.694267 8724 raft_consensus.cc:3037] T a54b2154ec784eaa84129bbf5d1b904e P e84d3c7d475a4ad9a1d40d8938b60cf7 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:10:44.694618 8747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:10:44.672863 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.235.66:40407 (local address 127.6.235.67:57335)
0928 02:10:44.685452 (+ 12589us) negotiation.cc:107] Waiting for socket to connect
0928 02:10:44.685480 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0928 02:10:44.685606 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:10:44.686180 (+ 574us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:10:44.686187 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:10:44.686213 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:10:44.686676 (+ 463us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:10:44.686688 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:10:44.688160 (+ 1472us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:10:44.688165 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:10:44.690223 (+ 2058us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:10:44.690234 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:10:44.692498 (+ 2264us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:10:44.692507 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:10:44.692711 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:10:44.692773 (+ 62us) client_negotiation.cc:769] Sending connection context
0928 02:10:44.693884 (+ 1111us) client_negotiation.cc:240] Negotiation successful
0928 02:10:44.694064 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12436,"mutex_wait_us":1,"thread_start_us":12241,"threads_started":1}
I20260928 02:10:44.695623 8513 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a54b2154ec784eaa84129bbf5d1b904e" candidate_uuid: "e84d3c7d475a4ad9a1d40d8938b60cf7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "931a82ffd0ef48a48003604170d99a22" is_pre_election: true
I20260928 02:10:44.696197 8513 raft_consensus.cc:2454] T a54b2154ec784eaa84129bbf5d1b904e P 931a82ffd0ef48a48003604170d99a22 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e84d3c7d475a4ad9a1d40d8938b60cf7 in term 0.
I20260928 02:10:44.697319 8724 raft_consensus.cc:504] T a54b2154ec784eaa84129bbf5d1b904e P e84d3c7d475a4ad9a1d40d8938b60cf7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1296711f54774aa2880bc8d1d2ec7b70" member_type: VOTER last_known_addr { host: "127.6.235.65" port: 34341 } } peers { permanent_uuid: "931a82ffd0ef48a48003604170d99a22" member_type: VOTER last_known_addr { host: "127.6.235.66" port: 40407 } } peers { permanent_uuid: "e84d3c7d475a4ad9a1d40d8938b60cf7" member_type: VOTER last_known_addr { host: "127.6.235.67" port: 42213 } }
I20260928 02:10:44.698372 8513 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a54b2154ec784eaa84129bbf5d1b904e" candidate_uuid: "e84d3c7d475a4ad9a1d40d8938b60cf7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "931a82ffd0ef48a48003604170d99a22"
I20260928 02:10:44.698565 8513 raft_consensus.cc:3037] T a54b2154ec784eaa84129bbf5d1b904e P 931a82ffd0ef48a48003604170d99a22 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:10:44.701172 8369 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a54b2154ec784eaa84129bbf5d1b904e" candidate_uuid: "e84d3c7d475a4ad9a1d40d8938b60cf7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1296711f54774aa2880bc8d1d2ec7b70"
I20260928 02:10:44.701380 8369 raft_consensus.cc:3037] T a54b2154ec784eaa84129bbf5d1b904e P 1296711f54774aa2880bc8d1d2ec7b70 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:10:44.701877 8513 raft_consensus.cc:2454] T a54b2154ec784eaa84129bbf5d1b904e P 931a82ffd0ef48a48003604170d99a22 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e84d3c7d475a4ad9a1d40d8938b60cf7 in term 1.
I20260928 02:10:44.702294 8580 leader_election.cc:302] T a54b2154ec784eaa84129bbf5d1b904e P e84d3c7d475a4ad9a1d40d8938b60cf7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 931a82ffd0ef48a48003604170d99a22, e84d3c7d475a4ad9a1d40d8938b60cf7; no voters:
I20260928 02:10:44.704401 8369 raft_consensus.cc:2454] T a54b2154ec784eaa84129bbf5d1b904e P 1296711f54774aa2880bc8d1d2ec7b70 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e84d3c7d475a4ad9a1d40d8938b60cf7 in term 1.
I20260928 02:10:44.705039 8724 leader_election.cc:288] T a54b2154ec784eaa84129bbf5d1b904e P e84d3c7d475a4ad9a1d40d8938b60cf7 [CANDIDATE]: Term 1 election: Requested vote from peers 1296711f54774aa2880bc8d1d2ec7b70 (127.6.235.65:34341), 931a82ffd0ef48a48003604170d99a22 (127.6.235.66:40407)
I20260928 02:10:44.705295 8724 raft_consensus.cc:2781] T a54b2154ec784eaa84129bbf5d1b904e P e84d3c7d475a4ad9a1d40d8938b60cf7 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:10:44.705538 8724 raft_consensus.cc:686] T a54b2154ec784eaa84129bbf5d1b904e P e84d3c7d475a4ad9a1d40d8938b60cf7 [term 1 LEADER]: Becoming Leader. State: Replica: e84d3c7d475a4ad9a1d40d8938b60cf7, State: Running, Role: LEADER
I20260928 02:10:44.706022 8724 consensus_queue.cc:237] T a54b2154ec784eaa84129bbf5d1b904e P e84d3c7d475a4ad9a1d40d8938b60cf7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1296711f54774aa2880bc8d1d2ec7b70" member_type: VOTER last_known_addr { host: "127.6.235.65" port: 34341 } } peers { permanent_uuid: "931a82ffd0ef48a48003604170d99a22" member_type: VOTER last_known_addr { host: "127.6.235.66" port: 40407 } } peers { permanent_uuid: "e84d3c7d475a4ad9a1d40d8938b60cf7" member_type: VOTER last_known_addr { host: "127.6.235.67" port: 42213 } }
I20260928 02:10:44.711174 8211 catalog_manager.cc:5495] T a54b2154ec784eaa84129bbf5d1b904e P e84d3c7d475a4ad9a1d40d8938b60cf7 reported cstate change: term changed from 0 to 1, leader changed from <none> to e84d3c7d475a4ad9a1d40d8938b60cf7 (127.6.235.67). New cstate: current_term: 1 leader_uuid: "e84d3c7d475a4ad9a1d40d8938b60cf7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1296711f54774aa2880bc8d1d2ec7b70" member_type: VOTER last_known_addr { host: "127.6.235.65" port: 34341 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "931a82ffd0ef48a48003604170d99a22" member_type: VOTER last_known_addr { host: "127.6.235.66" port: 40407 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e84d3c7d475a4ad9a1d40d8938b60cf7" member_type: VOTER last_known_addr { host: "127.6.235.67" port: 42213 } health_report { overall_health: HEALTHY } } }
Sep 28 02:10:44 dist-test-slave-86gv krb5kdc[8167](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561442, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.235.126@KRBTEST.COM for kudu/127.6.235.126@KRBTEST.COM
I20260928 02:10:44.755086 8306 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:10:44.739654 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46616 (local address 127.6.235.126:37431)
0928 02:10:44.739882 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0928 02:10:44.739893 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 02:10:44.740544 (+ 651us) server_negotiation.cc:407] Connection header received
0928 02:10:44.740688 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:10:44.740694 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:10:44.740794 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:10:44.740940 (+ 146us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:10:44.742310 (+ 1370us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:10:44.743488 (+ 1178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:10:44.744791 (+ 1303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:10:44.745525 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:10:44.745653 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:10:44.749269 (+ 3616us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:10:44.749274 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:10:44.749277 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:10:44.749304 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:10:44.752176 (+ 2872us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:10:44.753009 (+ 833us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:10:44.753013 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:10:44.753015 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:10:44.753152 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:10:44.753572 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:10:44.753577 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:10:44.753579 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:10:44.753869 (+ 290us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:10:44.753985 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0928 02:10:44.754477 (+ 492us) server_negotiation.cc:299] Negotiation successful
0928 02:10:44.754815 (+ 338us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":157,"server-negotiator.queue_time_us":66}
I20260928 02:10:44.755172 8755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:10:44.739735 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.235.126:37431 (local address 127.0.0.1:46616)
0928 02:10:44.740419 (+ 684us) negotiation.cc:107] Waiting for socket to connect
0928 02:10:44.740440 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0928 02:10:44.740572 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:10:44.741061 (+ 489us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:10:44.741065 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:10:44.741827 (+ 762us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:10:44.742157 (+ 330us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:10:44.742167 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:10:44.743634 (+ 1467us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:10:44.743639 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:10:44.744631 (+ 992us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:10:44.744644 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:10:44.745703 (+ 1059us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:10:44.745708 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:10:44.745837 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:10:44.746599 (+ 762us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:10:44.746616 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:10:44.749096 (+ 2480us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:10:44.752331 (+ 3235us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:10:44.752344 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:10:44.752351 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:10:44.752865 (+ 514us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:10:44.753257 (+ 392us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:10:44.753261 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:10:44.753264 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:10:44.753423 (+ 159us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:10:44.754020 (+ 597us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:10:44.754028 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:10:44.754209 (+ 181us) client_negotiation.cc:769] Sending connection context
0928 02:10:44.754377 (+ 168us) client_negotiation.cc:240] Negotiation successful
0928 02:10:44.754674 (+ 297us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":520,"thread_start_us":367,"threads_started":1}
I20260928 02:10:44.760612 8211 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.6.235.126@KRBTEST.COM'} at 127.0.0.1:46616:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260928 02:10:44.761876 8211 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 02:10:44.769285 8348 tablet_service.cc:1467] Processing CreateTablet for tablet f1ee96e50c8a4716ae17c14d8810736d (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=d043492b08994653832d2ec87e187c16]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260928 02:10:44.769784 8348 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f1ee96e50c8a4716ae17c14d8810736d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:10:44.775084 8627 tablet_service.cc:1467] Processing CreateTablet for tablet f1ee96e50c8a4716ae17c14d8810736d (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=d043492b08994653832d2ec87e187c16]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260928 02:10:44.775089 8488 tablet_service.cc:1467] Processing CreateTablet for tablet f1ee96e50c8a4716ae17c14d8810736d (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=d043492b08994653832d2ec87e187c16]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260928 02:10:44.775646 8627 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f1ee96e50c8a4716ae17c14d8810736d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:10:44.775661 8488 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f1ee96e50c8a4716ae17c14d8810736d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:10:44.782495 8717 tablet_bootstrap.cc:492] T f1ee96e50c8a4716ae17c14d8810736d P 1296711f54774aa2880bc8d1d2ec7b70: Bootstrap starting.
I20260928 02:10:44.783236 8718 tablet_bootstrap.cc:492] T f1ee96e50c8a4716ae17c14d8810736d P e84d3c7d475a4ad9a1d40d8938b60cf7: Bootstrap starting.
I20260928 02:10:44.784476 8717 tablet_bootstrap.cc:654] T f1ee96e50c8a4716ae17c14d8810736d P 1296711f54774aa2880bc8d1d2ec7b70: Neither blocks nor log segments found. Creating new log.
I20260928 02:10:44.785126 8718 tablet_bootstrap.cc:654] T f1ee96e50c8a4716ae17c14d8810736d P e84d3c7d475a4ad9a1d40d8938b60cf7: Neither blocks nor log segments found. Creating new log.
I20260928 02:10:44.785229 8716 tablet_bootstrap.cc:492] T f1ee96e50c8a4716ae17c14d8810736d P 931a82ffd0ef48a48003604170d99a22: Bootstrap starting.
I20260928 02:10:44.787036 8716 tablet_bootstrap.cc:654] T f1ee96e50c8a4716ae17c14d8810736d P 931a82ffd0ef48a48003604170d99a22: Neither blocks nor log segments found. Creating new log.
I20260928 02:10:44.794234 8716 tablet_bootstrap.cc:492] T f1ee96e50c8a4716ae17c14d8810736d P 931a82ffd0ef48a48003604170d99a22: No bootstrap required, opened a new log
I20260928 02:10:44.794503 8716 ts_tablet_manager.cc:1397] T f1ee96e50c8a4716ae17c14d8810736d P 931a82ffd0ef48a48003604170d99a22: Time spent bootstrapping tablet: real 0.009s user 0.004s sys 0.002s
I20260928 02:10:44.795650 8716 raft_consensus.cc:348] T f1ee96e50c8a4716ae17c14d8810736d P 931a82ffd0ef48a48003604170d99a22 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1296711f54774aa2880bc8d1d2ec7b70" member_type: VOTER last_known_addr { host: "127.6.235.65" port: 34341 } } peers { permanent_uuid: "931a82ffd0ef48a48003604170d99a22" member_type: VOTER last_known_addr { host: "127.6.235.66" port: 40407 } } peers { permanent_uuid: "e84d3c7d475a4ad9a1d40d8938b60cf7" member_type: VOTER last_known_addr { host: "127.6.235.67" port: 42213 } }
I20260928 02:10:44.796020 8716 raft_consensus.cc:374] T f1ee96e50c8a4716ae17c14d8810736d P 931a82ffd0ef48a48003604170d99a22 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:10:44.796131 8716 raft_consensus.cc:729] T f1ee96e50c8a4716ae17c14d8810736d P 931a82ffd0ef48a48003604170d99a22 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 931a82ffd0ef48a48003604170d99a22, State: Initialized, Role: FOLLOWER
I20260928 02:10:44.796522 8716 consensus_queue.cc:260] T f1ee96e50c8a4716ae17c14d8810736d P 931a82ffd0ef48a48003604170d99a22 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1296711f54774aa2880bc8d1d2ec7b70" member_type: VOTER last_known_addr { host: "127.6.235.65" port: 34341 } } peers { permanent_uuid: "931a82ffd0ef48a48003604170d99a22" member_type: VOTER last_known_addr { host: "127.6.235.66" port: 40407 } } peers { permanent_uuid: "e84d3c7d475a4ad9a1d40d8938b60cf7" member_type: VOTER last_known_addr { host: "127.6.235.67" port: 42213 } }
I20260928 02:10:44.797195 8735 tablet_replica.cc:439] T f1ee96e50c8a4716ae17c14d8810736d P 931a82ffd0ef48a48003604170d99a22: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1296711f54774aa2880bc8d1d2ec7b70" member_type: VOTER last_known_addr { host: "127.6.235.65" port: 34341 } } peers { permanent_uuid: "931a82ffd0ef48a48003604170d99a22" member_type: VOTER last_known_addr { host: "127.6.235.66" port: 40407 } } peers { permanent_uuid: "e84d3c7d475a4ad9a1d40d8938b60cf7" member_type: VOTER last_known_addr { host: "127.6.235.67" port: 42213 } } }
I20260928 02:10:44.797720 8735 tablet_replica.cc:442] T f1ee96e50c8a4716ae17c14d8810736d P 931a82ffd0ef48a48003604170d99a22: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 02:10:44.798914 8718 tablet_bootstrap.cc:492] T f1ee96e50c8a4716ae17c14d8810736d P e84d3c7d475a4ad9a1d40d8938b60cf7: No bootstrap required, opened a new log
I20260928 02:10:44.799376 8718 ts_tablet_manager.cc:1397] T f1ee96e50c8a4716ae17c14d8810736d P e84d3c7d475a4ad9a1d40d8938b60cf7: Time spent bootstrapping tablet: real 0.016s user 0.001s sys 0.012s
I20260928 02:10:44.800226 8718 raft_consensus.cc:348] T f1ee96e50c8a4716ae17c14d8810736d P e84d3c7d475a4ad9a1d40d8938b60cf7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1296711f54774aa2880bc8d1d2ec7b70" member_type: VOTER last_known_addr { host: "127.6.235.65" port: 34341 } } peers { permanent_uuid: "931a82ffd0ef48a48003604170d99a22" member_type: VOTER last_known_addr { host: "127.6.235.66" port: 40407 } } peers { permanent_uuid: "e84d3c7d475a4ad9a1d40d8938b60cf7" member_type: VOTER last_known_addr { host: "127.6.235.67" port: 42213 } }
I20260928 02:10:44.800537 8718 raft_consensus.cc:374] T f1ee96e50c8a4716ae17c14d8810736d P e84d3c7d475a4ad9a1d40d8938b60cf7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:10:44.800653 8718 raft_consensus.cc:729] T f1ee96e50c8a4716ae17c14d8810736d P e84d3c7d475a4ad9a1d40d8938b60cf7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e84d3c7d475a4ad9a1d40d8938b60cf7, State: Initialized, Role: FOLLOWER
I20260928 02:10:44.800992 8718 consensus_queue.cc:260] T f1ee96e50c8a4716ae17c14d8810736d P e84d3c7d475a4ad9a1d40d8938b60cf7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1296711f54774aa2880bc8d1d2ec7b70" member_type: VOTER last_known_addr { host: "127.6.235.65" port: 34341 } } peers { permanent_uuid: "931a82ffd0ef48a48003604170d99a22" member_type: VOTER last_known_addr { host: "127.6.235.66" port: 40407 } } peers { permanent_uuid: "e84d3c7d475a4ad9a1d40d8938b60cf7" member_type: VOTER last_known_addr { host: "127.6.235.67" port: 42213 } }
I20260928 02:10:44.801632 8716 ts_tablet_manager.cc:1428] T f1ee96e50c8a4716ae17c14d8810736d P 931a82ffd0ef48a48003604170d99a22: Time spent starting tablet: real 0.007s user 0.000s sys 0.004s
I20260928 02:10:44.801687 8724 tablet_replica.cc:439] T f1ee96e50c8a4716ae17c14d8810736d P e84d3c7d475a4ad9a1d40d8938b60cf7: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1296711f54774aa2880bc8d1d2ec7b70" member_type: VOTER last_known_addr { host: "127.6.235.65" port: 34341 } } peers { permanent_uuid: "931a82ffd0ef48a48003604170d99a22" member_type: VOTER last_known_addr { host: "127.6.235.66" port: 40407 } } peers { permanent_uuid: "e84d3c7d475a4ad9a1d40d8938b60cf7" member_type: VOTER last_known_addr { host: "127.6.235.67" port: 42213 } } }
I20260928 02:10:44.801970 8724 tablet_replica.cc:442] T f1ee96e50c8a4716ae17c14d8810736d P e84d3c7d475a4ad9a1d40d8938b60cf7: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 02:10:44.802469 8718 ts_tablet_manager.cc:1428] T f1ee96e50c8a4716ae17c14d8810736d P e84d3c7d475a4ad9a1d40d8938b60cf7: Time spent starting tablet: real 0.003s user 0.000s sys 0.001s
I20260928 02:10:44.804556 8717 tablet_bootstrap.cc:492] T f1ee96e50c8a4716ae17c14d8810736d P 1296711f54774aa2880bc8d1d2ec7b70: No bootstrap required, opened a new log
I20260928 02:10:44.804728 8717 ts_tablet_manager.cc:1397] T f1ee96e50c8a4716ae17c14d8810736d P 1296711f54774aa2880bc8d1d2ec7b70: Time spent bootstrapping tablet: real 0.022s user 0.003s sys 0.016s
I20260928 02:10:44.805554 8717 raft_consensus.cc:348] T f1ee96e50c8a4716ae17c14d8810736d P 1296711f54774aa2880bc8d1d2ec7b70 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1296711f54774aa2880bc8d1d2ec7b70" member_type: VOTER last_known_addr { host: "127.6.235.65" port: 34341 } } peers { permanent_uuid: "931a82ffd0ef48a48003604170d99a22" member_type: VOTER last_known_addr { host: "127.6.235.66" port: 40407 } } peers { permanent_uuid: "e84d3c7d475a4ad9a1d40d8938b60cf7" member_type: VOTER last_known_addr { host: "127.6.235.67" port: 42213 } }
I20260928 02:10:44.805850 8717 raft_consensus.cc:374] T f1ee96e50c8a4716ae17c14d8810736d P 1296711f54774aa2880bc8d1d2ec7b70 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:10:44.805981 8717 raft_consensus.cc:729] T f1ee96e50c8a4716ae17c14d8810736d P 1296711f54774aa2880bc8d1d2ec7b70 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1296711f54774aa2880bc8d1d2ec7b70, State: Initialized, Role: FOLLOWER
I20260928 02:10:44.806222 8717 consensus_queue.cc:260] T f1ee96e50c8a4716ae17c14d8810736d P 1296711f54774aa2880bc8d1d2ec7b70 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1296711f54774aa2880bc8d1d2ec7b70" member_type: VOTER last_known_addr { host: "127.6.235.65" port: 34341 } } peers { permanent_uuid: "931a82ffd0ef48a48003604170d99a22" member_type: VOTER last_known_addr { host: "127.6.235.66" port: 40407 } } peers { permanent_uuid: "e84d3c7d475a4ad9a1d40d8938b60cf7" member_type: VOTER last_known_addr { host: "127.6.235.67" port: 42213 } }
I20260928 02:10:44.806862 8717 ts_tablet_manager.cc:1428] T f1ee96e50c8a4716ae17c14d8810736d P 1296711f54774aa2880bc8d1d2ec7b70: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20260928 02:10:44.806739 8723 tablet_replica.cc:439] T f1ee96e50c8a4716ae17c14d8810736d P 1296711f54774aa2880bc8d1d2ec7b70: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1296711f54774aa2880bc8d1d2ec7b70" member_type: VOTER last_known_addr { host: "127.6.235.65" port: 34341 } } peers { permanent_uuid: "931a82ffd0ef48a48003604170d99a22" member_type: VOTER last_known_addr { host: "127.6.235.66" port: 40407 } } peers { permanent_uuid: "e84d3c7d475a4ad9a1d40d8938b60cf7" member_type: VOTER last_known_addr { host: "127.6.235.67" port: 42213 } } }
I20260928 02:10:44.807154 8723 tablet_replica.cc:442] T f1ee96e50c8a4716ae17c14d8810736d P 1296711f54774aa2880bc8d1d2ec7b70: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 02:10:44.811734 8735 raft_consensus.cc:482] T f1ee96e50c8a4716ae17c14d8810736d P 931a82ffd0ef48a48003604170d99a22 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 02:10:44.811977 8735 raft_consensus.cc:504] T f1ee96e50c8a4716ae17c14d8810736d P 931a82ffd0ef48a48003604170d99a22 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1296711f54774aa2880bc8d1d2ec7b70" member_type: VOTER last_known_addr { host: "127.6.235.65" port: 34341 } } peers { permanent_uuid: "931a82ffd0ef48a48003604170d99a22" member_type: VOTER last_known_addr { host: "127.6.235.66" port: 40407 } } peers { permanent_uuid: "e84d3c7d475a4ad9a1d40d8938b60cf7" member_type: VOTER last_known_addr { host: "127.6.235.67" port: 42213 } }
I20260928 02:10:44.812842 8735 leader_election.cc:288] T f1ee96e50c8a4716ae17c14d8810736d P 931a82ffd0ef48a48003604170d99a22 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1296711f54774aa2880bc8d1d2ec7b70 (127.6.235.65:34341), e84d3c7d475a4ad9a1d40d8938b60cf7 (127.6.235.67:42213)
I20260928 02:10:44.813320 8369 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f1ee96e50c8a4716ae17c14d8810736d" candidate_uuid: "931a82ffd0ef48a48003604170d99a22" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1296711f54774aa2880bc8d1d2ec7b70" is_pre_election: true
I20260928 02:10:44.813370 8647 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f1ee96e50c8a4716ae17c14d8810736d" candidate_uuid: "931a82ffd0ef48a48003604170d99a22" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e84d3c7d475a4ad9a1d40d8938b60cf7" is_pre_election: true
I20260928 02:10:44.813567 8369 raft_consensus.cc:2454] T f1ee96e50c8a4716ae17c14d8810736d P 1296711f54774aa2880bc8d1d2ec7b70 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 931a82ffd0ef48a48003604170d99a22 in term 0.
I20260928 02:10:44.813629 8647 raft_consensus.cc:2454] T f1ee96e50c8a4716ae17c14d8810736d P e84d3c7d475a4ad9a1d40d8938b60cf7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 931a82ffd0ef48a48003604170d99a22 in term 0.
I20260928 02:10:44.814138 8441 leader_election.cc:302] T f1ee96e50c8a4716ae17c14d8810736d P 931a82ffd0ef48a48003604170d99a22 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 1296711f54774aa2880bc8d1d2ec7b70, 931a82ffd0ef48a48003604170d99a22, e84d3c7d475a4ad9a1d40d8938b60cf7; no voters:
I20260928 02:10:44.814486 8735 raft_consensus.cc:2781] T f1ee96e50c8a4716ae17c14d8810736d P 931a82ffd0ef48a48003604170d99a22 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 02:10:44.814668 8735 raft_consensus.cc:482] T f1ee96e50c8a4716ae17c14d8810736d P 931a82ffd0ef48a48003604170d99a22 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 02:10:44.814788 8735 raft_consensus.cc:3037] T f1ee96e50c8a4716ae17c14d8810736d P 931a82ffd0ef48a48003604170d99a22 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:10:44.817406 8735 raft_consensus.cc:504] T f1ee96e50c8a4716ae17c14d8810736d P 931a82ffd0ef48a48003604170d99a22 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1296711f54774aa2880bc8d1d2ec7b70" member_type: VOTER last_known_addr { host: "127.6.235.65" port: 34341 } } peers { permanent_uuid: "931a82ffd0ef48a48003604170d99a22" member_type: VOTER last_known_addr { host: "127.6.235.66" port: 40407 } } peers { permanent_uuid: "e84d3c7d475a4ad9a1d40d8938b60cf7" member_type: VOTER last_known_addr { host: "127.6.235.67" port: 42213 } }
I20260928 02:10:44.818041 8735 leader_election.cc:288] T f1ee96e50c8a4716ae17c14d8810736d P 931a82ffd0ef48a48003604170d99a22 [CANDIDATE]: Term 1 election: Requested vote from peers 1296711f54774aa2880bc8d1d2ec7b70 (127.6.235.65:34341), e84d3c7d475a4ad9a1d40d8938b60cf7 (127.6.235.67:42213)
I20260928 02:10:44.818414 8369 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f1ee96e50c8a4716ae17c14d8810736d" candidate_uuid: "931a82ffd0ef48a48003604170d99a22" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1296711f54774aa2880bc8d1d2ec7b70"
I20260928 02:10:44.818645 8369 raft_consensus.cc:3037] T f1ee96e50c8a4716ae17c14d8810736d P 1296711f54774aa2880bc8d1d2ec7b70 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:10:44.818655 8647 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f1ee96e50c8a4716ae17c14d8810736d" candidate_uuid: "931a82ffd0ef48a48003604170d99a22" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e84d3c7d475a4ad9a1d40d8938b60cf7"
I20260928 02:10:44.818845 8647 raft_consensus.cc:3037] T f1ee96e50c8a4716ae17c14d8810736d P e84d3c7d475a4ad9a1d40d8938b60cf7 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:10:44.821897 8369 raft_consensus.cc:2454] T f1ee96e50c8a4716ae17c14d8810736d P 1296711f54774aa2880bc8d1d2ec7b70 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 931a82ffd0ef48a48003604170d99a22 in term 1.
I20260928 02:10:44.822173 8647 raft_consensus.cc:2454] T f1ee96e50c8a4716ae17c14d8810736d P e84d3c7d475a4ad9a1d40d8938b60cf7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 931a82ffd0ef48a48003604170d99a22 in term 1.
I20260928 02:10:44.822706 8441 leader_election.cc:302] T f1ee96e50c8a4716ae17c14d8810736d P 931a82ffd0ef48a48003604170d99a22 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 931a82ffd0ef48a48003604170d99a22, e84d3c7d475a4ad9a1d40d8938b60cf7; no voters:
I20260928 02:10:44.823088 8735 raft_consensus.cc:2781] T f1ee96e50c8a4716ae17c14d8810736d P 931a82ffd0ef48a48003604170d99a22 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:10:44.823272 8735 raft_consensus.cc:686] T f1ee96e50c8a4716ae17c14d8810736d P 931a82ffd0ef48a48003604170d99a22 [term 1 LEADER]: Becoming Leader. State: Replica: 931a82ffd0ef48a48003604170d99a22, State: Running, Role: LEADER
I20260928 02:10:44.823571 8735 consensus_queue.cc:237] T f1ee96e50c8a4716ae17c14d8810736d P 931a82ffd0ef48a48003604170d99a22 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1296711f54774aa2880bc8d1d2ec7b70" member_type: VOTER last_known_addr { host: "127.6.235.65" port: 34341 } } peers { permanent_uuid: "931a82ffd0ef48a48003604170d99a22" member_type: VOTER last_known_addr { host: "127.6.235.66" port: 40407 } } peers { permanent_uuid: "e84d3c7d475a4ad9a1d40d8938b60cf7" member_type: VOTER last_known_addr { host: "127.6.235.67" port: 42213 } }
I20260928 02:10:44.825455 8727 tablet_replica.cc:439] T f1ee96e50c8a4716ae17c14d8810736d P 931a82ffd0ef48a48003604170d99a22: TxnStatusTablet state changed. Reason: New leader 931a82ffd0ef48a48003604170d99a22. Latest consensus state: current_term: 1 leader_uuid: "931a82ffd0ef48a48003604170d99a22" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1296711f54774aa2880bc8d1d2ec7b70" member_type: VOTER last_known_addr { host: "127.6.235.65" port: 34341 } } peers { permanent_uuid: "931a82ffd0ef48a48003604170d99a22" member_type: VOTER last_known_addr { host: "127.6.235.66" port: 40407 } } peers { permanent_uuid: "e84d3c7d475a4ad9a1d40d8938b60cf7" member_type: VOTER last_known_addr { host: "127.6.235.67" port: 42213 } } }
I20260928 02:10:44.825819 8727 tablet_replica.cc:442] T f1ee96e50c8a4716ae17c14d8810736d P 931a82ffd0ef48a48003604170d99a22: This TxnStatusTablet replica's current role is: LEADER
I20260928 02:10:44.826614 8760 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260928 02:10:44.826853 8760 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260928 02:10:44.827049 8211 catalog_manager.cc:5495] T f1ee96e50c8a4716ae17c14d8810736d P 931a82ffd0ef48a48003604170d99a22 reported cstate change: term changed from 0 to 1, leader changed from <none> to 931a82ffd0ef48a48003604170d99a22 (127.6.235.66). New cstate: current_term: 1 leader_uuid: "931a82ffd0ef48a48003604170d99a22" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1296711f54774aa2880bc8d1d2ec7b70" member_type: VOTER last_known_addr { host: "127.6.235.65" port: 34341 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "931a82ffd0ef48a48003604170d99a22" member_type: VOTER last_known_addr { host: "127.6.235.66" port: 40407 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e84d3c7d475a4ad9a1d40d8938b60cf7" member_type: VOTER last_known_addr { host: "127.6.235.67" port: 42213 } health_report { overall_health: UNKNOWN } } }
I20260928 02:10:44.849677 8755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:10:44.843569 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.235.66:40407 (local address 127.0.0.1:34864)
0928 02:10:44.843822 (+ 253us) negotiation.cc:107] Waiting for socket to connect
0928 02:10:44.843844 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0928 02:10:44.843977 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:10:44.844468 (+ 491us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:10:44.844472 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:10:44.844488 (+ 16us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 02:10:44.844664 (+ 176us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:10:44.844673 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:10:44.846301 (+ 1628us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:10:44.846306 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:10:44.847426 (+ 1120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:10:44.847439 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:10:44.848485 (+ 1046us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:10:44.848489 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:10:44.848639 (+ 150us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:10:44.848700 (+ 61us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 02:10:44.849228 (+ 528us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 02:10:44.849255 (+ 27us) client_negotiation.cc:769] Sending connection context
0928 02:10:44.849330 (+ 75us) client_negotiation.cc:240] Negotiation successful
0928 02:10:44.849457 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":59}
I20260928 02:10:44.849745 8715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:10:44.843789 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34864 (local address 127.6.235.66:40407)
0928 02:10:44.844017 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0928 02:10:44.844023 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 02:10:44.844056 (+ 33us) server_negotiation.cc:407] Connection header received
0928 02:10:44.844131 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:10:44.844137 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:10:44.844233 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:10:44.844343 (+ 110us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 02:10:44.844845 (+ 502us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:10:44.846144 (+ 1299us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:10:44.847583 (+ 1439us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:10:44.848292 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:10:44.848434 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:10:44.848956 (+ 522us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 02:10:44.849114 (+ 158us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 02:10:44.849213 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0928 02:10:44.849380 (+ 167us) server_negotiation.cc:299] Negotiation successful
0928 02:10:44.849474 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260928 02:10:44.853746 8369 raft_consensus.cc:1261] T f1ee96e50c8a4716ae17c14d8810736d P 1296711f54774aa2880bc8d1d2ec7b70 [term 1 FOLLOWER]: Refusing update from remote peer 931a82ffd0ef48a48003604170d99a22: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 02:10:44.853827 8647 raft_consensus.cc:1261] T f1ee96e50c8a4716ae17c14d8810736d P e84d3c7d475a4ad9a1d40d8938b60cf7 [term 1 FOLLOWER]: Refusing update from remote peer 931a82ffd0ef48a48003604170d99a22: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 02:10:44.854562 8735 consensus_queue.cc:1035] T f1ee96e50c8a4716ae17c14d8810736d P 931a82ffd0ef48a48003604170d99a22 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1296711f54774aa2880bc8d1d2ec7b70" member_type: VOTER last_known_addr { host: "127.6.235.65" port: 34341 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 02:10:44.854907 8727 consensus_queue.cc:1035] T f1ee96e50c8a4716ae17c14d8810736d P 931a82ffd0ef48a48003604170d99a22 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e84d3c7d475a4ad9a1d40d8938b60cf7" member_type: VOTER last_known_addr { host: "127.6.235.67" port: 42213 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 02:10:44.857681 8724 tablet_replica.cc:439] T f1ee96e50c8a4716ae17c14d8810736d P e84d3c7d475a4ad9a1d40d8938b60cf7: TxnStatusTablet state changed. Reason: New leader 931a82ffd0ef48a48003604170d99a22. Latest consensus state: current_term: 1 leader_uuid: "931a82ffd0ef48a48003604170d99a22" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1296711f54774aa2880bc8d1d2ec7b70" member_type: VOTER last_known_addr { host: "127.6.235.65" port: 34341 } } peers { permanent_uuid: "931a82ffd0ef48a48003604170d99a22" member_type: VOTER last_known_addr { host: "127.6.235.66" port: 40407 } } peers { permanent_uuid: "e84d3c7d475a4ad9a1d40d8938b60cf7" member_type: VOTER last_known_addr { host: "127.6.235.67" port: 42213 } } }
I20260928 02:10:44.857995 8724 tablet_replica.cc:442] T f1ee96e50c8a4716ae17c14d8810736d P e84d3c7d475a4ad9a1d40d8938b60cf7: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 02:10:44.861835 8724 tablet_replica.cc:439] T f1ee96e50c8a4716ae17c14d8810736d P e84d3c7d475a4ad9a1d40d8938b60cf7: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "931a82ffd0ef48a48003604170d99a22" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1296711f54774aa2880bc8d1d2ec7b70" member_type: VOTER last_known_addr { host: "127.6.235.65" port: 34341 } } peers { permanent_uuid: "931a82ffd0ef48a48003604170d99a22" member_type: VOTER last_known_addr { host: "127.6.235.66" port: 40407 } } peers { permanent_uuid: "e84d3c7d475a4ad9a1d40d8938b60cf7" member_type: VOTER last_known_addr { host: "127.6.235.67" port: 42213 } } }
I20260928 02:10:44.862143 8724 tablet_replica.cc:442] T f1ee96e50c8a4716ae17c14d8810736d P e84d3c7d475a4ad9a1d40d8938b60cf7: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 02:10:44.865559 8722 tablet_replica.cc:439] T f1ee96e50c8a4716ae17c14d8810736d P 931a82ffd0ef48a48003604170d99a22: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "931a82ffd0ef48a48003604170d99a22" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1296711f54774aa2880bc8d1d2ec7b70" member_type: VOTER last_known_addr { host: "127.6.235.65" port: 34341 } } peers { permanent_uuid: "931a82ffd0ef48a48003604170d99a22" member_type: VOTER last_known_addr { host: "127.6.235.66" port: 40407 } } peers { permanent_uuid: "e84d3c7d475a4ad9a1d40d8938b60cf7" member_type: VOTER last_known_addr { host: "127.6.235.67" port: 42213 } } }
I20260928 02:10:44.865916 8722 tablet_replica.cc:442] T f1ee96e50c8a4716ae17c14d8810736d P 931a82ffd0ef48a48003604170d99a22: This TxnStatusTablet replica's current role is: LEADER
I20260928 02:10:44.869854 8723 tablet_replica.cc:439] T f1ee96e50c8a4716ae17c14d8810736d P 1296711f54774aa2880bc8d1d2ec7b70: TxnStatusTablet state changed. Reason: New leader 931a82ffd0ef48a48003604170d99a22. Latest consensus state: current_term: 1 leader_uuid: "931a82ffd0ef48a48003604170d99a22" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1296711f54774aa2880bc8d1d2ec7b70" member_type: VOTER last_known_addr { host: "127.6.235.65" port: 34341 } } peers { permanent_uuid: "931a82ffd0ef48a48003604170d99a22" member_type: VOTER last_known_addr { host: "127.6.235.66" port: 40407 } } peers { permanent_uuid: "e84d3c7d475a4ad9a1d40d8938b60cf7" member_type: VOTER last_known_addr { host: "127.6.235.67" port: 42213 } } }
I20260928 02:10:44.870133 8723 tablet_replica.cc:442] T f1ee96e50c8a4716ae17c14d8810736d P 1296711f54774aa2880bc8d1d2ec7b70: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 02:10:44.870332 8735 tablet_replica.cc:439] T f1ee96e50c8a4716ae17c14d8810736d P 931a82ffd0ef48a48003604170d99a22: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "931a82ffd0ef48a48003604170d99a22" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1296711f54774aa2880bc8d1d2ec7b70" member_type: VOTER last_known_addr { host: "127.6.235.65" port: 34341 } } peers { permanent_uuid: "931a82ffd0ef48a48003604170d99a22" member_type: VOTER last_known_addr { host: "127.6.235.66" port: 40407 } } peers { permanent_uuid: "e84d3c7d475a4ad9a1d40d8938b60cf7" member_type: VOTER last_known_addr { host: "127.6.235.67" port: 42213 } } }
I20260928 02:10:44.870620 8735 tablet_replica.cc:442] T f1ee96e50c8a4716ae17c14d8810736d P 931a82ffd0ef48a48003604170d99a22: This TxnStatusTablet replica's current role is: LEADER
I20260928 02:10:44.873433 8723 tablet_replica.cc:439] T f1ee96e50c8a4716ae17c14d8810736d P 1296711f54774aa2880bc8d1d2ec7b70: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "931a82ffd0ef48a48003604170d99a22" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1296711f54774aa2880bc8d1d2ec7b70" member_type: VOTER last_known_addr { host: "127.6.235.65" port: 34341 } } peers { permanent_uuid: "931a82ffd0ef48a48003604170d99a22" member_type: VOTER last_known_addr { host: "127.6.235.66" port: 40407 } } peers { permanent_uuid: "e84d3c7d475a4ad9a1d40d8938b60cf7" member_type: VOTER last_known_addr { host: "127.6.235.67" port: 42213 } } }
I20260928 02:10:44.873668 8723 tablet_replica.cc:442] T f1ee96e50c8a4716ae17c14d8810736d P 1296711f54774aa2880bc8d1d2ec7b70: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 02:10:44.880127 8713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:10:44.873408 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45700 (local address 127.6.235.67:42213)
0928 02:10:44.873607 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0928 02:10:44.873613 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 02:10:44.873632 (+ 19us) server_negotiation.cc:407] Connection header received
0928 02:10:44.873741 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:10:44.873746 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:10:44.873895 (+ 149us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:10:44.874092 (+ 197us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 02:10:44.874588 (+ 496us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:10:44.875791 (+ 1203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:10:44.877349 (+ 1558us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:10:44.877964 (+ 615us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:10:44.878118 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:10:44.878587 (+ 469us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 02:10:44.879427 (+ 840us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 02:10:44.879592 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0928 02:10:44.879702 (+ 110us) server_negotiation.cc:299] Negotiation successful
0928 02:10:44.879798 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260928 02:10:44.896984 8715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:10:44.891173 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.235.67:39755 (local address 127.6.235.66:40407)
0928 02:10:44.891503 (+ 330us) server_negotiation.cc:206] Beginning negotiation
0928 02:10:44.891516 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0928 02:10:44.891975 (+ 459us) server_negotiation.cc:407] Connection header received
0928 02:10:44.892205 (+ 230us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:10:44.892212 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:10:44.892277 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:10:44.892384 (+ 107us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 02:10:44.892823 (+ 439us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:10:44.893744 (+ 921us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:10:44.895130 (+ 1386us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:10:44.895609 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:10:44.895752 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:10:44.896182 (+ 430us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 02:10:44.896302 (+ 120us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 02:10:44.896409 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0928 02:10:44.896549 (+ 140us) server_negotiation.cc:299] Negotiation successful
0928 02:10:44.896624 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260928 02:10:44.897068 8766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:10:44.891160 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.235.66:40407 (local address 127.6.235.67:39755)
0928 02:10:44.891867 (+ 707us) negotiation.cc:107] Waiting for socket to connect
0928 02:10:44.891903 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0928 02:10:44.892064 (+ 161us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:10:44.892400 (+ 336us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:10:44.892404 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:10:44.892415 (+ 11us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 02:10:44.892688 (+ 273us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:10:44.892696 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:10:44.893878 (+ 1182us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:10:44.893886 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:10:44.894971 (+ 1085us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:10:44.894983 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:10:44.895786 (+ 803us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:10:44.895793 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:10:44.895929 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:10:44.895987 (+ 58us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 02:10:44.896422 (+ 435us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 02:10:44.896435 (+ 13us) client_negotiation.cc:769] Sending connection context
0928 02:10:44.896529 (+ 94us) client_negotiation.cc:240] Negotiation successful
0928 02:10:44.896663 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":532,"thread_start_us":386,"threads_started":1}
I20260928 02:10:44.908239 8369 raft_consensus.cc:1261] T a54b2154ec784eaa84129bbf5d1b904e P 1296711f54774aa2880bc8d1d2ec7b70 [term 1 FOLLOWER]: Refusing update from remote peer e84d3c7d475a4ad9a1d40d8938b60cf7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 02:10:44.908486 8513 raft_consensus.cc:1261] T a54b2154ec784eaa84129bbf5d1b904e P 931a82ffd0ef48a48003604170d99a22 [term 1 FOLLOWER]: Refusing update from remote peer e84d3c7d475a4ad9a1d40d8938b60cf7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 02:10:44.909175 8748 consensus_queue.cc:1035] T a54b2154ec784eaa84129bbf5d1b904e P e84d3c7d475a4ad9a1d40d8938b60cf7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1296711f54774aa2880bc8d1d2ec7b70" member_type: VOTER last_known_addr { host: "127.6.235.65" port: 34341 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 02:10:44.909602 8724 consensus_queue.cc:1035] T a54b2154ec784eaa84129bbf5d1b904e P e84d3c7d475a4ad9a1d40d8938b60cf7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "931a82ffd0ef48a48003604170d99a22" member_type: VOTER last_known_addr { host: "127.6.235.66" port: 40407 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 02:10:44.946746 8713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:10:44.939842 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.235.66:53585 (local address 127.6.235.67:42213)
0928 02:10:44.940157 (+ 315us) server_negotiation.cc:206] Beginning negotiation
0928 02:10:44.940165 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 02:10:44.940785 (+ 620us) server_negotiation.cc:407] Connection header received
0928 02:10:44.941041 (+ 256us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:10:44.941046 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:10:44.941126 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:10:44.941225 (+ 99us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 02:10:44.941817 (+ 592us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:10:44.942874 (+ 1057us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:10:44.944585 (+ 1711us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:10:44.945322 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:10:44.945463 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:10:44.946002 (+ 539us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 02:10:44.946125 (+ 123us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 02:10:44.946224 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0928 02:10:44.946364 (+ 140us) server_negotiation.cc:299] Negotiation successful
0928 02:10:44.946448 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":177}
I20260928 02:10:44.947510 8772 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:10:44.939417 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.235.67:42213 (local address 127.6.235.66:53585)
0928 02:10:44.940676 (+ 1259us) negotiation.cc:107] Waiting for socket to connect
0928 02:10:44.940692 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 02:10:44.940862 (+ 170us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:10:44.941239 (+ 377us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:10:44.941244 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:10:44.941260 (+ 16us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 02:10:44.941617 (+ 357us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:10:44.941649 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:10:44.943005 (+ 1356us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:10:44.943010 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:10:44.944435 (+ 1425us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:10:44.944448 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:10:44.945549 (+ 1101us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:10:44.945556 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:10:44.945739 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:10:44.945826 (+ 87us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 02:10:44.946240 (+ 414us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 02:10:44.946252 (+ 12us) client_negotiation.cc:769] Sending connection context
0928 02:10:44.946331 (+ 79us) client_negotiation.cc:240] Negotiation successful
0928 02:10:44.946479 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1067,"thread_start_us":828,"threads_started":1}
I20260928 02:10:44.992228 8212 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:46612:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 02:10:44.992496 8212 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:46612:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 02:10:44.996186 8212 catalog_manager.cc:5782] T 00000000000000000000000000000000 P ae6ca031c9744a50b87bd00e3e3df444: Sending DeleteTablet for 3 replicas of tablet a54b2154ec784eaa84129bbf5d1b904e
I20260928 02:10:44.997105 8348 tablet_service.cc:1514] Processing DeleteTablet for tablet a54b2154ec784eaa84129bbf5d1b904e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:10:44 UTC) from {username='kudu', principal='kudu/127.6.235.126@KRBTEST.COM'} at 127.0.0.1:47860
I20260928 02:10:44.997159 8627 tablet_service.cc:1514] Processing DeleteTablet for tablet a54b2154ec784eaa84129bbf5d1b904e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:10:44 UTC) from {username='kudu', principal='kudu/127.6.235.126@KRBTEST.COM'} at 127.0.0.1:45692
I20260928 02:10:44.997396 8740 tablet_replica.cc:330] T a54b2154ec784eaa84129bbf5d1b904e P 1296711f54774aa2880bc8d1d2ec7b70: stopping tablet replica
I20260928 02:10:44.997447 8741 tablet_replica.cc:330] T a54b2154ec784eaa84129bbf5d1b904e P e84d3c7d475a4ad9a1d40d8938b60cf7: stopping tablet replica
I20260928 02:10:44.997611 8740 raft_consensus.cc:2229] T a54b2154ec784eaa84129bbf5d1b904e P 1296711f54774aa2880bc8d1d2ec7b70 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 02:10:44.997643 8741 raft_consensus.cc:2229] T a54b2154ec784eaa84129bbf5d1b904e P e84d3c7d475a4ad9a1d40d8938b60cf7 [term 1 LEADER]: Raft consensus shutting down.
I20260928 02:10:44.997813 8740 raft_consensus.cc:2258] T a54b2154ec784eaa84129bbf5d1b904e P 1296711f54774aa2880bc8d1d2ec7b70 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 02:10:44.998179 8741 raft_consensus.cc:2258] T a54b2154ec784eaa84129bbf5d1b904e P e84d3c7d475a4ad9a1d40d8938b60cf7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 02:10:44.999434 8741 ts_tablet_manager.cc:1905] T a54b2154ec784eaa84129bbf5d1b904e P e84d3c7d475a4ad9a1d40d8938b60cf7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 02:10:44.999908 8740 ts_tablet_manager.cc:1905] T a54b2154ec784eaa84129bbf5d1b904e P 1296711f54774aa2880bc8d1d2ec7b70: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 02:10:45.006714 8741 ts_tablet_manager.cc:1918] T a54b2154ec784eaa84129bbf5d1b904e P e84d3c7d475a4ad9a1d40d8938b60cf7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 02:10:45.006881 8741 log.cc:1196] T a54b2154ec784eaa84129bbf5d1b904e P e84d3c7d475a4ad9a1d40d8938b60cf7: Deleting WAL directory at /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790561432251112-7085-0/minicluster-data/ts-2/wal/wals/a54b2154ec784eaa84129bbf5d1b904e
I20260928 02:10:45.006811 8488 tablet_service.cc:1514] Processing DeleteTablet for tablet a54b2154ec784eaa84129bbf5d1b904e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:10:44 UTC) from {username='kudu', principal='kudu/127.6.235.126@KRBTEST.COM'} at 127.0.0.1:34854
I20260928 02:10:45.007145 8742 tablet_replica.cc:330] T a54b2154ec784eaa84129bbf5d1b904e P 931a82ffd0ef48a48003604170d99a22: stopping tablet replica
I20260928 02:10:45.007309 8741 ts_tablet_manager.cc:1939] T a54b2154ec784eaa84129bbf5d1b904e P e84d3c7d475a4ad9a1d40d8938b60cf7: Deleting consensus metadata
I20260928 02:10:45.007354 8742 raft_consensus.cc:2229] T a54b2154ec784eaa84129bbf5d1b904e P 931a82ffd0ef48a48003604170d99a22 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 02:10:45.007584 8742 raft_consensus.cc:2258] T a54b2154ec784eaa84129bbf5d1b904e P 931a82ffd0ef48a48003604170d99a22 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 02:10:45.008883 8742 ts_tablet_manager.cc:1905] T a54b2154ec784eaa84129bbf5d1b904e P 931a82ffd0ef48a48003604170d99a22: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 02:10:45.012754 8740 ts_tablet_manager.cc:1918] T a54b2154ec784eaa84129bbf5d1b904e P 1296711f54774aa2880bc8d1d2ec7b70: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 02:10:45.014181 8740 log.cc:1196] T a54b2154ec784eaa84129bbf5d1b904e P 1296711f54774aa2880bc8d1d2ec7b70: Deleting WAL directory at /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790561432251112-7085-0/minicluster-data/ts-0/wal/wals/a54b2154ec784eaa84129bbf5d1b904e
I20260928 02:10:45.015300 8740 ts_tablet_manager.cc:1939] T a54b2154ec784eaa84129bbf5d1b904e P 1296711f54774aa2880bc8d1d2ec7b70: Deleting consensus metadata
I20260928 02:10:45.016162 8742 ts_tablet_manager.cc:1918] T a54b2154ec784eaa84129bbf5d1b904e P 931a82ffd0ef48a48003604170d99a22: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 02:10:45.016340 8742 log.cc:1196] T a54b2154ec784eaa84129bbf5d1b904e P 931a82ffd0ef48a48003604170d99a22: Deleting WAL directory at /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790561432251112-7085-0/minicluster-data/ts-1/wal/wals/a54b2154ec784eaa84129bbf5d1b904e
I20260928 02:10:45.016767 8742 ts_tablet_manager.cc:1939] T a54b2154ec784eaa84129bbf5d1b904e P 931a82ffd0ef48a48003604170d99a22: Deleting consensus metadata
I20260928 02:10:45.017555 8197 catalog_manager.cc:4841] TS 1296711f54774aa2880bc8d1d2ec7b70 (127.6.235.65:34341): tablet a54b2154ec784eaa84129bbf5d1b904e (table test-table [id=f60e0dc0b21a41009f53d4f6e7dedbf1]) successfully deleted
I20260928 02:10:45.017890 8198 catalog_manager.cc:4841] TS 931a82ffd0ef48a48003604170d99a22 (127.6.235.66:40407): tablet a54b2154ec784eaa84129bbf5d1b904e (table test-table [id=f60e0dc0b21a41009f53d4f6e7dedbf1]) successfully deleted
I20260928 02:10:45.018778 8198 catalog_manager.cc:4841] TS e84d3c7d475a4ad9a1d40d8938b60cf7 (127.6.235.67:42213): tablet a54b2154ec784eaa84129bbf5d1b904e (table test-table [id=f60e0dc0b21a41009f53d4f6e7dedbf1]) successfully deleted
Sep 28 02:10:45 dist-test-slave-86gv krb5kdc[8167](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561444, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.6.235.65@KRBTEST.COM
I20260928 02:10:45.044373 8712 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:10:45.023285 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47866 (local address 127.6.235.65:34341)
0928 02:10:45.023466 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0928 02:10:45.023473 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 02:10:45.024087 (+ 614us) server_negotiation.cc:407] Connection header received
0928 02:10:45.024375 (+ 288us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:10:45.024382 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:10:45.024473 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:10:45.024585 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:10:45.026575 (+ 1990us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:10:45.027653 (+ 1078us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:10:45.029635 (+ 1982us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:10:45.030280 (+ 645us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:10:45.030407 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:10:45.035769 (+ 5362us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:10:45.035790 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:10:45.035812 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:10:45.035861 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:10:45.040191 (+ 4330us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:10:45.041110 (+ 919us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:10:45.041121 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:10:45.041126 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:10:45.041307 (+ 181us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:10:45.041854 (+ 547us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:10:45.041860 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:10:45.041863 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:10:45.042319 (+ 456us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:10:45.042629 (+ 310us) server_negotiation.cc:1035] Waiting for connection context
0928 02:10:45.043789 (+ 1160us) server_negotiation.cc:299] Negotiation successful
0928 02:10:45.044006 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
W20260928 02:10:45.045521 8414 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.SetFlag from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:47866
I20260928 02:10:45.072144 8306 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:10:45.049266 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46628 (local address 127.6.235.126:37431)
0928 02:10:45.049465 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0928 02:10:45.049471 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 02:10:45.055923 (+ 6452us) server_negotiation.cc:407] Connection header received
0928 02:10:45.056130 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:10:45.056136 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:10:45.056219 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:10:45.056291 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:10:45.058245 (+ 1954us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:10:45.059380 (+ 1135us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:10:45.061326 (+ 1946us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:10:45.062005 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:10:45.062137 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:10:45.065247 (+ 3110us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:10:45.065255 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:10:45.065260 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:10:45.065307 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:10:45.069006 (+ 3699us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:10:45.069931 (+ 925us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:10:45.069939 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:10:45.069943 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:10:45.070108 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:10:45.070535 (+ 427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:10:45.070540 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:10:45.070544 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:10:45.070933 (+ 389us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:10:45.071209 (+ 276us) server_negotiation.cc:1035] Waiting for connection context
0928 02:10:45.071542 (+ 333us) server_negotiation.cc:299] Negotiation successful
0928 02:10:45.071778 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
W20260928 02:10:45.072738 8212 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:46628
I20260928 02:10:45.075101 7085 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskTlaUuc/build/asan/bin/kudu pid 8283
I20260928 02:10:45.097716 8712 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:10:45.076997 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47880 (local address 127.6.235.65:34341)
0928 02:10:45.077189 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0928 02:10:45.077195 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 02:10:45.081767 (+ 4572us) server_negotiation.cc:407] Connection header received
0928 02:10:45.081940 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:10:45.081946 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:10:45.082034 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:10:45.082147 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:10:45.083752 (+ 1605us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:10:45.084940 (+ 1188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:10:45.086779 (+ 1839us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:10:45.087456 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:10:45.087603 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:10:45.090646 (+ 3043us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:10:45.090654 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:10:45.090660 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:10:45.090707 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:10:45.094444 (+ 3737us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:10:45.095347 (+ 903us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:10:45.095353 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:10:45.095357 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:10:45.095524 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:10:45.095943 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:10:45.095949 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:10:45.095953 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:10:45.096283 (+ 330us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:10:45.096575 (+ 292us) server_negotiation.cc:1035] Waiting for connection context
0928 02:10:45.096846 (+ 271us) server_negotiation.cc:299] Negotiation successful
0928 02:10:45.097126 (+ 280us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
W20260928 02:10:45.098270 8414 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:47880
W20260928 02:10:45.098937 7085 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskTlaUuc/build/asan/bin/kudu pid 8283: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260928 02:10:45.099131 7085 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTlaUuc/build/asan/bin/kudu with pid 8283
I20260928 02:10:45.125960 7085 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskTlaUuc/build/asan/bin/kudu pid 8423
Sep 28 02:10:45 dist-test-slave-86gv krb5kdc[8167](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561444, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.6.235.66@KRBTEST.COM
I20260928 02:10:45.146556 8715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:10:45.126762 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34880 (local address 127.6.235.66:40407)
0928 02:10:45.126977 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0928 02:10:45.126983 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 02:10:45.127269 (+ 286us) server_negotiation.cc:407] Connection header received
0928 02:10:45.127435 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:10:45.127441 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:10:45.127536 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:10:45.127660 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:10:45.129254 (+ 1594us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:10:45.130509 (+ 1255us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:10:45.132353 (+ 1844us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:10:45.133077 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:10:45.133223 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:10:45.138755 (+ 5532us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:10:45.138775 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:10:45.138797 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:10:45.138844 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:10:45.143043 (+ 4199us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:10:45.143898 (+ 855us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:10:45.143907 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:10:45.143911 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:10:45.144066 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:10:45.144476 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:10:45.144481 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:10:45.144492 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:10:45.145141 (+ 649us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:10:45.145463 (+ 322us) server_negotiation.cc:1035] Waiting for connection context
0928 02:10:45.145748 (+ 285us) server_negotiation.cc:299] Negotiation successful
0928 02:10:45.146000 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
W20260928 02:10:45.147336 8553 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:34880
W20260928 02:10:45.148042 7085 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskTlaUuc/build/asan/bin/kudu pid 8423: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260928 02:10:45.148224 7085 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTlaUuc/build/asan/bin/kudu with pid 8423
I20260928 02:10:45.177757 7085 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskTlaUuc/build/asan/bin/kudu pid 8562
Sep 28 02:10:45 dist-test-slave-86gv krb5kdc[8167](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561444, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.6.235.67@KRBTEST.COM
I20260928 02:10:45.199379 8713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:10:45.178578 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45702 (local address 127.6.235.67:42213)
0928 02:10:45.178775 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0928 02:10:45.178788 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0928 02:10:45.178892 (+ 104us) server_negotiation.cc:407] Connection header received
0928 02:10:45.179082 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:10:45.179086 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:10:45.179154 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:10:45.179232 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:10:45.180963 (+ 1731us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:10:45.182274 (+ 1311us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:10:45.184214 (+ 1940us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:10:45.185014 (+ 800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:10:45.185142 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:10:45.191152 (+ 6010us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:10:45.191168 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:10:45.191188 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:10:45.191239 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:10:45.195751 (+ 4512us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:10:45.196811 (+ 1060us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:10:45.196821 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:10:45.196825 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:10:45.197088 (+ 263us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:10:45.197539 (+ 451us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:10:45.197544 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:10:45.197548 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:10:45.198132 (+ 584us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:10:45.198457 (+ 325us) server_negotiation.cc:1035] Waiting for connection context
0928 02:10:45.198743 (+ 286us) server_negotiation.cc:299] Negotiation successful
0928 02:10:45.198986 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
W20260928 02:10:45.200102 8692 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:45702
W20260928 02:10:45.200846 7085 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskTlaUuc/build/asan/bin/kudu pid 8562: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260928 02:10:45.201154 7085 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTlaUuc/build/asan/bin/kudu with pid 8562
I20260928 02:10:45.220986 7085 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskTlaUuc/build/asan/bin/kudu pid 8180
I20260928 02:10:45.222187 8252 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 02:10:45.339185 7085 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTlaUuc/build/asan/bin/kudu with pid 8180
2026-09-28T02:10:45Z chronyd exiting
[ OK ] SecurityITest.SmokeTestAsAuthorizedUser (4631 ms)
[ RUN ] SecurityITest.TxnSmokeWithDifferentUserTypes
Loading random data
Initializing database '/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790561432251112-7085-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 02:10:45 dist-test-slave-86gv krb5kdc[8784](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 02:10:45 dist-test-slave-86gv krb5kdc[8784](info): set up 2 sockets
Sep 28 02:10:45 dist-test-slave-86gv krb5kdc[8784](info): commencing operation
krb5kdc: starting...
W20260928 02:10:46.449466 7085 mini_kdc.cc:121] Time spent starting KDC: real 1.065s user 0.003s sys 0.015s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790561432251112-7085-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790561432251112-7085-0/krb5kdc/test-user.keytab.
Sep 28 02:10:46 dist-test-slave-86gv krb5kdc[8784](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561446, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T02:10:46Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T02:10:46Z Disabled control of system clock
WARNING: no policy specified for kudu/127.6.235.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.235.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.235.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790561432251112-7085-0/krb5kdc/kudu_127.6.235.126.keytab.
Entry for principal kudu/127.6.235.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790561432251112-7085-0/krb5kdc/kudu_127.6.235.126.keytab.
I20260928 02:10:46.648569 7085 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTlaUuc/build/asan/bin/kudu
/tmp/dist-test-taskTlaUuc/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790561432251112-7085-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790561432251112-7085-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790561432251112-7085-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790561432251112-7085-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.235.126:40759
--webserver_interface=127.6.235.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790561432251112-7085-0/krb5kdc/kudu_127.6.235.126.keytab
--principal=kudu/127.6.235.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.235.84:39741
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.235.126:40759
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790561432251112-7085-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790561432251112-7085-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790561432251112-7085-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:10:46.981156 8797 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:10:46.981539 8797 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:10:46.981696 8797 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:10:46.989135 8797 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 02:10:46.989308 8797 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260928 02:10:46.989408 8797 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:10:46.989537 8797 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:10:46.989624 8797 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 02:10:46.989717 8797 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 02:10:46.999854 8797 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.235.84:39741
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790561432251112-7085-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790561432251112-7085-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.235.126:40759
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790561432251112-7085-0/krb5kdc/kudu_127.6.235.126.keytab
--principal=kudu/127.6.235.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.235.126:40759
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790561432251112-7085-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.235.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790561432251112-7085-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 00:51:59 UTC on ccycloud.quasar-axgkwn.root.comops.site
build id 4676464
ASAN enabled
I20260928 02:10:47.001279 8797 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:10:47.002626 8797 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:10:47.014945 8802 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:10:47.015070 8803 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:10:47.014978 8805 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:10:47.015677 8797 server_base.cc:1034] running on GCE node
I20260928 02:10:47.016541 8797 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:10:47.018152 8797 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:10:47.019416 8797 hybrid_clock.cc:648] HybridClock initialized: now 1790561447019346 us; error 46 us; skew 500 ppm
I20260928 02:10:47.019770 8797 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 02:10:47 dist-test-slave-86gv krb5kdc[8784](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561447, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.235.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:10:47.023053 8797 init.cc:377] Logged in from keytab as kudu/127.6.235.126@KRBTEST.COM (short username kudu)
I20260928 02:10:47.024667 8797 webserver.cc:460] Webserver started at http://127.6.235.126:34313/ using document root <none> and password file <none>
I20260928 02:10:47.025470 8797 fs_manager.cc:359] Metadata directory not provided
I20260928 02:10:47.025619 8797 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:10:47.025938 8797 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:10:47.029158 8797 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790561432251112-7085-0/minicluster-data/master-0/data/instance:
uuid: "d52b1a21d6d74a8a8e0b264bdb19a5ab"
format_stamp: "Formatted at 2026-09-28 02:10:47 on dist-test-slave-86gv"
server_key: "1a6f422bd6bbbac66cfe7b402ab87a80"
server_key_iv: "485adf625df8848438b774c57baa2718"
server_key_version: "encryptionkey@0"
I20260928 02:10:47.029937 8797 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790561432251112-7085-0/minicluster-data/master-0/wal/instance:
uuid: "d52b1a21d6d74a8a8e0b264bdb19a5ab"
format_stamp: "Formatted at 2026-09-28 02:10:47 on dist-test-slave-86gv"
server_key: "1a6f422bd6bbbac66cfe7b402ab87a80"
server_key_iv: "485adf625df8848438b774c57baa2718"
server_key_version: "encryptionkey@0"
I20260928 02:10:47.035362 8797 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.002s
I20260928 02:10:47.039232 8812 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:10:47.040597 8797 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260928 02:10:47.040788 8797 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790561432251112-7085-0/minicluster-data/master-0/data,/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790561432251112-7085-0/minicluster-data/master-0/wal
uuid: "d52b1a21d6d74a8a8e0b264bdb19a5ab"
format_stamp: "Formatted at 2026-09-28 02:10:47 on dist-test-slave-86gv"
server_key: "1a6f422bd6bbbac66cfe7b402ab87a80"
server_key_iv: "485adf625df8848438b774c57baa2718"
server_key_version: "encryptionkey@0"
I20260928 02:10:47.041072 8797 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790561432251112-7085-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790561432251112-7085-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790561432251112-7085-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:10:47.071137 8797 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:10:47.075383 8797 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:10:47.075681 8797 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:10:47.097769 8797 rpc_server.cc:307] RPC server started. Bound to: 127.6.235.126:40759
I20260928 02:10:47.097793 8873 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.235.126:40759 every 8 connection(s)
I20260928 02:10:47.099195 8797 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790561432251112-7085-0/minicluster-data/master-0/data/info.pb
I20260928 02:10:47.100426 7085 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTlaUuc/build/asan/bin/kudu as pid 8797
I20260928 02:10:47.100652 7085 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790561432251112-7085-0/minicluster-data/master-0/wal/instance
I20260928 02:10:47.101399 7085 external_mini_cluster.cc:1428] Setting key 30456801fc9190ec46d4516a009250aa
I20260928 02:10:47.103780 8874 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 28 02:10:47 dist-test-slave-86gv krb5kdc[8784](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561446, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.235.126@KRBTEST.COM
I20260928 02:10:47.117331 8874 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d52b1a21d6d74a8a8e0b264bdb19a5ab: Bootstrap starting.
I20260928 02:10:47.121618 8874 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d52b1a21d6d74a8a8e0b264bdb19a5ab: Neither blocks nor log segments found. Creating new log.
I20260928 02:10:47.122992 8874 log.cc:824] T 00000000000000000000000000000000 P d52b1a21d6d74a8a8e0b264bdb19a5ab: Log is configured to *not* fsync() on all Append() calls
I20260928 02:10:47.125917 8877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:10:47.103367 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52588 (local address 127.6.235.126:40759)
0928 02:10:47.104148 (+ 781us) server_negotiation.cc:206] Beginning negotiation
0928 02:10:47.104161 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0928 02:10:47.104201 (+ 40us) server_negotiation.cc:407] Connection header received
0928 02:10:47.105970 (+ 1769us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:10:47.105993 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:10:47.106407 (+ 414us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:10:47.106768 (+ 361us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:10:47.108554 (+ 1786us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:10:47.109991 (+ 1437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:10:47.111716 (+ 1725us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:10:47.112358 (+ 642us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:10:47.112491 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:10:47.117180 (+ 4689us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:10:47.117191 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:10:47.117211 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:10:47.117256 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:10:47.121202 (+ 3946us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:10:47.122012 (+ 810us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:10:47.122025 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:10:47.122036 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:10:47.122247 (+ 211us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:10:47.122632 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:10:47.122639 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:10:47.122644 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:10:47.123385 (+ 741us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:10:47.123748 (+ 363us) server_negotiation.cc:1035] Waiting for connection context
0928 02:10:47.124212 (+ 464us) server_negotiation.cc:299] Negotiation successful
0928 02:10:47.124692 (+ 480us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":564,"thread_start_us":370,"threads_started":1}
I20260928 02:10:47.126309 8874 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d52b1a21d6d74a8a8e0b264bdb19a5ab: No bootstrap required, opened a new log
I20260928 02:10:47.134841 8874 raft_consensus.cc:348] T 00000000000000000000000000000000 P d52b1a21d6d74a8a8e0b264bdb19a5ab [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d52b1a21d6d74a8a8e0b264bdb19a5ab" member_type: VOTER last_known_addr { host: "127.6.235.126" port: 40759 } }
I20260928 02:10:47.135253 8874 raft_consensus.cc:374] T 00000000000000000000000000000000 P d52b1a21d6d74a8a8e0b264bdb19a5ab [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:10:47.135366 8874 raft_consensus.cc:729] T 00000000000000000000000000000000 P d52b1a21d6d74a8a8e0b264bdb19a5ab [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d52b1a21d6d74a8a8e0b264bdb19a5ab, State: Initialized, Role: FOLLOWER
I20260928 02:10:47.136020 8874 consensus_queue.cc:260] T 00000000000000000000000000000000 P d52b1a21d6d74a8a8e0b264bdb19a5ab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d52b1a21d6d74a8a8e0b264bdb19a5ab" member_type: VOTER last_known_addr { host: "127.6.235.126" port: 40759 } }
I20260928 02:10:47.136271 8874 raft_consensus.cc:388] T 00000000000000000000000000000000 P d52b1a21d6d74a8a8e0b264bdb19a5ab [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 02:10:47.136415 8874 raft_consensus.cc:482] T 00000000000000000000000000000000 P d52b1a21d6d74a8a8e0b264bdb19a5ab [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 02:10:47.136629 8874 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d52b1a21d6d74a8a8e0b264bdb19a5ab [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:10:47.139065 8874 raft_consensus.cc:504] T 00000000000000000000000000000000 P d52b1a21d6d74a8a8e0b264bdb19a5ab [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d52b1a21d6d74a8a8e0b264bdb19a5ab" member_type: VOTER last_known_addr { host: "127.6.235.126" port: 40759 } }
I20260928 02:10:47.139701 8874 leader_election.cc:302] T 00000000000000000000000000000000 P d52b1a21d6d74a8a8e0b264bdb19a5ab [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d52b1a21d6d74a8a8e0b264bdb19a5ab; no voters:
I20260928 02:10:47.140364 8874 leader_election.cc:288] T 00000000000000000000000000000000 P d52b1a21d6d74a8a8e0b264bdb19a5ab [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 02:10:47.140467 8879 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d52b1a21d6d74a8a8e0b264bdb19a5ab [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:10:47.141196 8879 raft_consensus.cc:686] T 00000000000000000000000000000000 P d52b1a21d6d74a8a8e0b264bdb19a5ab [term 1 LEADER]: Becoming Leader. State: Replica: d52b1a21d6d74a8a8e0b264bdb19a5ab, State: Running, Role: LEADER
I20260928 02:10:47.141706 8879 consensus_queue.cc:237] T 00000000000000000000000000000000 P d52b1a21d6d74a8a8e0b264bdb19a5ab [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d52b1a21d6d74a8a8e0b264bdb19a5ab" member_type: VOTER last_known_addr { host: "127.6.235.126" port: 40759 } }
I20260928 02:10:47.142506 8874 sys_catalog.cc:564] T 00000000000000000000000000000000 P d52b1a21d6d74a8a8e0b264bdb19a5ab [sys.catalog]: configured and running, proceeding with master startup.
I20260928 02:10:47.145100 8880 sys_catalog.cc:455] T 00000000000000000000000000000000 P d52b1a21d6d74a8a8e0b264bdb19a5ab [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d52b1a21d6d74a8a8e0b264bdb19a5ab" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d52b1a21d6d74a8a8e0b264bdb19a5ab" member_type: VOTER last_known_addr { host: "127.6.235.126" port: 40759 } } }
I20260928 02:10:47.145460 8880 sys_catalog.cc:458] T 00000000000000000000000000000000 P d52b1a21d6d74a8a8e0b264bdb19a5ab [sys.catalog]: This master's current role is: LEADER
I20260928 02:10:47.145797 8881 sys_catalog.cc:455] T 00000000000000000000000000000000 P d52b1a21d6d74a8a8e0b264bdb19a5ab [sys.catalog]: SysCatalogTable state changed. Reason: New leader d52b1a21d6d74a8a8e0b264bdb19a5ab. Latest consensus state: current_term: 1 leader_uuid: "d52b1a21d6d74a8a8e0b264bdb19a5ab" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d52b1a21d6d74a8a8e0b264bdb19a5ab" member_type: VOTER last_known_addr { host: "127.6.235.126" port: 40759 } } }
I20260928 02:10:47.146068 8881 sys_catalog.cc:458] T 00000000000000000000000000000000 P d52b1a21d6d74a8a8e0b264bdb19a5ab [sys.catalog]: This master's current role is: LEADER
I20260928 02:10:47.150517 8887 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 02:10:47.156526 8887 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 02:10:47.165467 8887 catalog_manager.cc:1347] Generated new cluster ID: d884796654774cb987de5cb8ea8aeeab
I20260928 02:10:47.165614 8887 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 02:10:47.199533 8887 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 02:10:47.200987 8887 catalog_manager.cc:1504] Loading token signing keys...
I20260928 02:10:47.206514 8887 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d52b1a21d6d74a8a8e0b264bdb19a5ab: Generated new TSK 0
I20260928 02:10:47.207417 8887 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.6.235.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.235.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.235.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790561432251112-7085-0/krb5kdc/kudu_127.6.235.65.keytab.
Entry for principal kudu/127.6.235.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790561432251112-7085-0/krb5kdc/kudu_127.6.235.65.keytab.
I20260928 02:10:47.261097 7085 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTlaUuc/build/asan/bin/kudu
/tmp/dist-test-taskTlaUuc/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790561432251112-7085-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790561432251112-7085-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790561432251112-7085-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790561432251112-7085-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.235.65:0
--local_ip_for_outbound_sockets=127.6.235.65
--webserver_interface=127.6.235.65
--webserver_port=0
--tserver_master_addrs=127.6.235.126:40759
--keytab_file=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790561432251112-7085-0/krb5kdc/kudu_127.6.235.65.keytab
--principal=kudu/127.6.235.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.235.84:39741
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790561432251112-7085-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790561432251112-7085-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790561432251112-7085-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:10:47.607178 8900 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:10:47.607488 8900 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:10:47.607656 8900 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:10:47.615094 8900 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:10:47.615213 8900 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:10:47.615384 8900 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260928 02:10:47.615466 8900 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260928 02:10:47.615597 8900 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.235.65
I20260928 02:10:47.625849 8900 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.235.84:39741
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790561432251112-7085-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790561432251112-7085-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790561432251112-7085-0/krb5kdc/kudu_127.6.235.65.keytab
--principal=kudu/127.6.235.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.235.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790561432251112-7085-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.6.235.65
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.6.235.126:40759
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.235.65
--log_dir=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790561432251112-7085-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 00:51:59 UTC on ccycloud.quasar-axgkwn.root.comops.site
build id 4676464
ASAN enabled
I20260928 02:10:47.627355 8900 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:10:47.628773 8900 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:10:47.643113 8905 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:10:47.643088 8906 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:10:47.643177 8908 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:10:47.643924 8900 server_base.cc:1034] running on GCE node
I20260928 02:10:47.644582 8900 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:10:47.645660 8900 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:10:47.646899 8900 hybrid_clock.cc:648] HybridClock initialized: now 1790561447646847 us; error 71 us; skew 500 ppm
I20260928 02:10:47.647248 8900 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 02:10:47 dist-test-slave-86gv krb5kdc[8784](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561447, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.235.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:10:47.650617 8900 init.cc:377] Logged in from keytab as kudu/127.6.235.65@KRBTEST.COM (short username kudu)
I20260928 02:10:47.652153 8900 webserver.cc:460] Webserver started at http://127.6.235.65:33913/ using document root <none> and password file <none>
I20260928 02:10:47.652839 8900 fs_manager.cc:359] Metadata directory not provided
I20260928 02:10:47.653030 8900 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:10:47.653352 8900 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:10:47.656378 8900 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790561432251112-7085-0/minicluster-data/ts-0/data/instance:
uuid: "1acdcdd55e604bc498460384fd246cb4"
format_stamp: "Formatted at 2026-09-28 02:10:47 on dist-test-slave-86gv"
server_key: "d3f8c73c64a1a5e1a9d492b97ef94beb"
server_key_iv: "2d96d986bc0a143a27c047559ede1c07"
server_key_version: "encryptionkey@0"
I20260928 02:10:47.657306 8900 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790561432251112-7085-0/minicluster-data/ts-0/wal/instance:
uuid: "1acdcdd55e604bc498460384fd246cb4"
format_stamp: "Formatted at 2026-09-28 02:10:47 on dist-test-slave-86gv"
server_key: "d3f8c73c64a1a5e1a9d492b97ef94beb"
server_key_iv: "2d96d986bc0a143a27c047559ede1c07"
server_key_version: "encryptionkey@0"
I20260928 02:10:47.662768 8900 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260928 02:10:47.666697 8915 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:10:47.668236 8900 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260928 02:10:47.668429 8900 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790561432251112-7085-0/minicluster-data/ts-0/data,/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790561432251112-7085-0/minicluster-data/ts-0/wal
uuid: "1acdcdd55e604bc498460384fd246cb4"
format_stamp: "Formatted at 2026-09-28 02:10:47 on dist-test-slave-86gv"
server_key: "d3f8c73c64a1a5e1a9d492b97ef94beb"
server_key_iv: "2d96d986bc0a143a27c047559ede1c07"
server_key_version: "encryptionkey@0"
I20260928 02:10:47.668651 8900 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790561432251112-7085-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790561432251112-7085-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790561432251112-7085-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:10:47.697937 8900 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:10:47.702281 8900 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:10:47.702601 8900 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:10:47.705436 8900 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:10:47.705569 8900 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:10:47.705686 8900 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:10:47.705813 8900 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 28 02:10:47 dist-test-slave-86gv krb5kdc[8784](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561447, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.235.65@KRBTEST.COM for kudu/127.6.235.126@KRBTEST.COM
I20260928 02:10:47.730775 8924 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:10:47.706273 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.235.65:56723 (local address 127.6.235.126:40759)
0928 02:10:47.706796 (+ 523us) server_negotiation.cc:206] Beginning negotiation
0928 02:10:47.706804 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 02:10:47.707826 (+ 1022us) server_negotiation.cc:407] Connection header received
0928 02:10:47.710022 (+ 2196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:10:47.710027 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:10:47.710115 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:10:47.710244 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:10:47.712683 (+ 2439us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:10:47.714070 (+ 1387us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:10:47.716634 (+ 2564us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:10:47.717264 (+ 630us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:10:47.717421 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:10:47.723183 (+ 5762us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:10:47.723192 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:10:47.723198 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:10:47.723251 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:10:47.726844 (+ 3593us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:10:47.727688 (+ 844us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:10:47.727694 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:10:47.727698 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:10:47.727826 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:10:47.728294 (+ 468us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:10:47.728301 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:10:47.728304 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:10:47.728595 (+ 291us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:10:47.728746 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0928 02:10:47.729885 (+ 1139us) server_negotiation.cc:299] Negotiation successful
0928 02:10:47.730277 (+ 392us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":348,"thread_start_us":217,"threads_started":1}
I20260928 02:10:47.731844 8925 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:10:47.706784 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.235.126:40759 (local address 127.6.235.65:56723)
0928 02:10:47.707642 (+ 858us) negotiation.cc:107] Waiting for socket to connect
0928 02:10:47.707690 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0928 02:10:47.709734 (+ 2044us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:10:47.710495 (+ 761us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:10:47.710504 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:10:47.711461 (+ 957us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:10:47.712418 (+ 957us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:10:47.712453 (+ 35us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:10:47.714266 (+ 1813us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:10:47.714274 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:10:47.716404 (+ 2130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:10:47.716419 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:10:47.717746 (+ 1327us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:10:47.717751 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:10:47.718097 (+ 346us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:10:47.718999 (+ 902us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:10:47.719039 (+ 40us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:10:47.722833 (+ 3794us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:10:47.727040 (+ 4207us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:10:47.727051 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:10:47.727072 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:10:47.727552 (+ 480us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:10:47.727947 (+ 395us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:10:47.727952 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:10:47.727956 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:10:47.728180 (+ 224us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:10:47.728744 (+ 564us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:10:47.728753 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:10:47.729358 (+ 605us) client_negotiation.cc:769] Sending connection context
0928 02:10:47.729732 (+ 374us) client_negotiation.cc:240] Negotiation successful
0928 02:10:47.730406 (+ 674us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":604,"thread_start_us":375,"threads_started":1}
I20260928 02:10:47.755892 8994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:10:47.738740 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.235.126:40759 (local address 127.6.235.65:59329)
0928 02:10:47.739401 (+ 661us) negotiation.cc:107] Waiting for socket to connect
0928 02:10:47.739424 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0928 02:10:47.739574 (+ 150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:10:47.740016 (+ 442us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:10:47.740020 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:10:47.740460 (+ 440us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:10:47.740741 (+ 281us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:10:47.740750 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:10:47.742379 (+ 1629us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:10:47.742385 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:10:47.744190 (+ 1805us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:10:47.744206 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:10:47.745464 (+ 1258us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:10:47.745471 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:10:47.745642 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:10:47.746418 (+ 776us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:10:47.746435 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:10:47.747178 (+ 743us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:10:47.751712 (+ 4534us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:10:47.751720 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:10:47.751726 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:10:47.752471 (+ 745us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:10:47.753016 (+ 545us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:10:47.753033 (+ 17us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:10:47.753039 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:10:47.753272 (+ 233us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:10:47.754169 (+ 897us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:10:47.754177 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:10:47.754470 (+ 293us) client_negotiation.cc:769] Sending connection context
0928 02:10:47.754668 (+ 198us) client_negotiation.cc:240] Negotiation successful
0928 02:10:47.755168 (+ 500us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":527,"thread_start_us":391,"threads_started":1}
I20260928 02:10:47.757140 8924 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:10:47.738180 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.235.65:59329 (local address 127.6.235.126:40759)
0928 02:10:47.738397 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0928 02:10:47.738402 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 02:10:47.739534 (+ 1132us) server_negotiation.cc:407] Connection header received
0928 02:10:47.739682 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:10:47.739690 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:10:47.739798 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:10:47.739939 (+ 141us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:10:47.740908 (+ 969us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:10:47.742185 (+ 1277us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:10:47.744387 (+ 2202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:10:47.745226 (+ 839us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:10:47.745385 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:10:47.747406 (+ 2021us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:10:47.747416 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:10:47.747423 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:10:47.747474 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:10:47.751519 (+ 4045us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:10:47.752622 (+ 1103us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:10:47.752630 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:10:47.752634 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:10:47.752831 (+ 197us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:10:47.753441 (+ 610us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:10:47.753450 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:10:47.753456 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:10:47.753916 (+ 460us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:10:47.754082 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0928 02:10:47.756302 (+ 2220us) server_negotiation.cc:299] Negotiation successful
0928 02:10:47.756621 (+ 319us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260928 02:10:47.759261 8900 rpc_server.cc:307] RPC server started. Bound to: 127.6.235.65:46109
I20260928 02:10:47.759312 9035 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.235.65:46109 every 8 connection(s)
I20260928 02:10:47.760826 8900 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790561432251112-7085-0/minicluster-data/ts-0/data/info.pb
I20260928 02:10:47.762346 7085 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTlaUuc/build/asan/bin/kudu as pid 8900
I20260928 02:10:47.762607 7085 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790561432251112-7085-0/minicluster-data/ts-0/wal/instance
I20260928 02:10:47.763310 7085 external_mini_cluster.cc:1428] Setting key f9d2ed164e8b8fcb83feb89354d361c1
I20260928 02:10:47.763185 9036 heartbeater.cc:347] Connected to a master server at 127.6.235.126:40759
I20260928 02:10:47.764437 9036 heartbeater.cc:464] Registering TS with master...
I20260928 02:10:47.765306 9036 heartbeater.cc:511] Master 127.6.235.126:40759 requested a full tablet report, sending...
I20260928 02:10:47.767385 8829 ts_manager.cc:194] Registered new tserver with Master: 1acdcdd55e604bc498460384fd246cb4 (127.6.235.65:46109)
I20260928 02:10:47.768955 8829 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.235.65@KRBTEST.COM'} at 127.6.235.65:56723
WARNING: no policy specified for kudu/127.6.235.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.235.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.235.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790561432251112-7085-0/krb5kdc/kudu_127.6.235.66.keytab.
Entry for principal kudu/127.6.235.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790561432251112-7085-0/krb5kdc/kudu_127.6.235.66.keytab.
I20260928 02:10:47.809067 7085 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTlaUuc/build/asan/bin/kudu
/tmp/dist-test-taskTlaUuc/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790561432251112-7085-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790561432251112-7085-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790561432251112-7085-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790561432251112-7085-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.235.66:0
--local_ip_for_outbound_sockets=127.6.235.66
--webserver_interface=127.6.235.66
--webserver_port=0
--tserver_master_addrs=127.6.235.126:40759
--keytab_file=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790561432251112-7085-0/krb5kdc/kudu_127.6.235.66.keytab
--principal=kudu/127.6.235.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.235.84:39741
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790561432251112-7085-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790561432251112-7085-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790561432251112-7085-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:10:48.147388 9040 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:10:48.147716 9040 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:10:48.147816 9040 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:10:48.155197 9040 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:10:48.155323 9040 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:10:48.155408 9040 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260928 02:10:48.155459 9040 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260928 02:10:48.155534 9040 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.235.66
I20260928 02:10:48.165568 9040 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.235.84:39741
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790561432251112-7085-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790561432251112-7085-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790561432251112-7085-0/krb5kdc/kudu_127.6.235.66.keytab
--principal=kudu/127.6.235.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.235.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790561432251112-7085-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.6.235.66
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.6.235.126:40759
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.235.66
--log_dir=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790561432251112-7085-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 00:51:59 UTC on ccycloud.quasar-axgkwn.root.comops.site
build id 4676464
ASAN enabled
I20260928 02:10:48.167032 9040 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:10:48.168395 9040 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:10:48.182097 9046 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:10:48.182066 9045 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:10:48.182170 9048 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:10:48.182919 9040 server_base.cc:1034] running on GCE node
I20260928 02:10:48.183584 9040 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:10:48.184636 9040 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:10:48.185962 9040 hybrid_clock.cc:648] HybridClock initialized: now 1790561448185883 us; error 63 us; skew 500 ppm
I20260928 02:10:48.186300 9040 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 02:10:48 dist-test-slave-86gv krb5kdc[8784](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561448, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.235.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:10:48.189864 9040 init.cc:377] Logged in from keytab as kudu/127.6.235.66@KRBTEST.COM (short username kudu)
I20260928 02:10:48.191440 9040 webserver.cc:460] Webserver started at http://127.6.235.66:42767/ using document root <none> and password file <none>
I20260928 02:10:48.192168 9040 fs_manager.cc:359] Metadata directory not provided
I20260928 02:10:48.192271 9040 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:10:48.192592 9040 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:10:48.195767 9040 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790561432251112-7085-0/minicluster-data/ts-1/data/instance:
uuid: "97e1a30f27ab4d1480891ee30a895fad"
format_stamp: "Formatted at 2026-09-28 02:10:48 on dist-test-slave-86gv"
server_key: "e22ebd88d6638b88bc58ef0436b2f1ba"
server_key_iv: "60ad8e02b3061e313bf0eee1af285044"
server_key_version: "encryptionkey@0"
I20260928 02:10:48.196728 9040 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790561432251112-7085-0/minicluster-data/ts-1/wal/instance:
uuid: "97e1a30f27ab4d1480891ee30a895fad"
format_stamp: "Formatted at 2026-09-28 02:10:48 on dist-test-slave-86gv"
server_key: "e22ebd88d6638b88bc58ef0436b2f1ba"
server_key_iv: "60ad8e02b3061e313bf0eee1af285044"
server_key_version: "encryptionkey@0"
I20260928 02:10:48.202638 9040 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260928 02:10:48.206559 9055 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:10:48.208122 9040 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.004s
I20260928 02:10:48.208307 9040 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790561432251112-7085-0/minicluster-data/ts-1/data,/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790561432251112-7085-0/minicluster-data/ts-1/wal
uuid: "97e1a30f27ab4d1480891ee30a895fad"
format_stamp: "Formatted at 2026-09-28 02:10:48 on dist-test-slave-86gv"
server_key: "e22ebd88d6638b88bc58ef0436b2f1ba"
server_key_iv: "60ad8e02b3061e313bf0eee1af285044"
server_key_version: "encryptionkey@0"
I20260928 02:10:48.208528 9040 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790561432251112-7085-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790561432251112-7085-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790561432251112-7085-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:10:48.222025 9040 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:10:48.226368 9040 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:10:48.226717 9040 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:10:48.229471 9040 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:10:48.229638 9040 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:10:48.229784 9040 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:10:48.229898 9040 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 28 02:10:48 dist-test-slave-86gv krb5kdc[8784](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561448, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.235.66@KRBTEST.COM for kudu/127.6.235.126@KRBTEST.COM
I20260928 02:10:48.258052 8924 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:10:48.230118 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.235.66:42269 (local address 127.6.235.126:40759)
0928 02:10:48.230365 (+ 247us) server_negotiation.cc:206] Beginning negotiation
0928 02:10:48.230372 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 02:10:48.231918 (+ 1546us) server_negotiation.cc:407] Connection header received
0928 02:10:48.235439 (+ 3521us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:10:48.235445 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:10:48.235590 (+ 145us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:10:48.235719 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:10:48.238861 (+ 3142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:10:48.239588 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:10:48.243550 (+ 3962us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:10:48.244176 (+ 626us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:10:48.244321 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:10:48.249524 (+ 5203us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:10:48.249529 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:10:48.249532 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:10:48.249566 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:10:48.254112 (+ 4546us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:10:48.255209 (+ 1097us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:10:48.255217 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:10:48.255223 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:10:48.255423 (+ 200us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:10:48.255898 (+ 475us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:10:48.255906 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:10:48.255913 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:10:48.256476 (+ 563us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:10:48.256700 (+ 224us) server_negotiation.cc:1035] Waiting for connection context
0928 02:10:48.257479 (+ 779us) server_negotiation.cc:299] Negotiation successful
0928 02:10:48.257698 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":90}
I20260928 02:10:48.259284 9064 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:10:48.230763 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.235.126:40759 (local address 127.6.235.66:42269)
0928 02:10:48.231714 (+ 951us) negotiation.cc:107] Waiting for socket to connect
0928 02:10:48.231770 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0928 02:10:48.234954 (+ 3184us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:10:48.236083 (+ 1129us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:10:48.236100 (+ 17us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:10:48.237343 (+ 1243us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:10:48.238615 (+ 1272us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:10:48.238655 (+ 40us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:10:48.239804 (+ 1149us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:10:48.239814 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:10:48.243323 (+ 3509us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:10:48.243362 (+ 39us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:10:48.244709 (+ 1347us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:10:48.244715 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:10:48.244998 (+ 283us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:10:48.245860 (+ 862us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:10:48.245904 (+ 44us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:10:48.249219 (+ 3315us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:10:48.254324 (+ 5105us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:10:48.254336 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:10:48.254361 (+ 25us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:10:48.255037 (+ 676us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:10:48.255561 (+ 524us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:10:48.255566 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:10:48.255568 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:10:48.255763 (+ 195us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:10:48.256627 (+ 864us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:10:48.256634 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:10:48.257079 (+ 445us) client_negotiation.cc:769] Sending connection context
0928 02:10:48.257377 (+ 298us) client_negotiation.cc:240] Negotiation successful
0928 02:10:48.257952 (+ 575us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":610,"thread_start_us":380,"threads_started":1}
I20260928 02:10:48.284409 8924 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:10:48.266895 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.235.66:57373 (local address 127.6.235.126:40759)
0928 02:10:48.267128 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0928 02:10:48.267136 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 02:10:48.268070 (+ 934us) server_negotiation.cc:407] Connection header received
0928 02:10:48.268282 (+ 212us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:10:48.268285 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:10:48.268348 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:10:48.268493 (+ 145us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:10:48.269928 (+ 1435us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:10:48.270967 (+ 1039us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:10:48.273946 (+ 2979us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:10:48.274579 (+ 633us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:10:48.274734 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:10:48.277692 (+ 2958us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:10:48.277697 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:10:48.277700 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:10:48.277731 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:10:48.280963 (+ 3232us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:10:48.282172 (+ 1209us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:10:48.282183 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:10:48.282186 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:10:48.282316 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:10:48.282832 (+ 516us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:10:48.282840 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:10:48.282845 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:10:48.283230 (+ 385us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:10:48.283388 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0928 02:10:48.283814 (+ 426us) server_negotiation.cc:299] Negotiation successful
0928 02:10:48.284048 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260928 02:10:48.284574 9135 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:10:48.266580 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.235.126:40759 (local address 127.6.235.66:57373)
0928 02:10:48.267910 (+ 1330us) negotiation.cc:107] Waiting for socket to connect
0928 02:10:48.267941 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0928 02:10:48.268148 (+ 207us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:10:48.268502 (+ 354us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:10:48.268507 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:10:48.269270 (+ 763us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:10:48.269749 (+ 479us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:10:48.269788 (+ 39us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:10:48.271163 (+ 1375us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:10:48.271173 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:10:48.273733 (+ 2560us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:10:48.273755 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:10:48.274846 (+ 1091us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:10:48.274856 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:10:48.275135 (+ 279us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:10:48.275955 (+ 820us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:10:48.275990 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:10:48.277501 (+ 1511us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:10:48.281134 (+ 3633us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:10:48.281147 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:10:48.281154 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:10:48.282013 (+ 859us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:10:48.282449 (+ 436us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:10:48.282458 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:10:48.282465 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:10:48.282679 (+ 214us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:10:48.283358 (+ 679us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:10:48.283363 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:10:48.283563 (+ 200us) client_negotiation.cc:769] Sending connection context
0928 02:10:48.283722 (+ 159us) client_negotiation.cc:240] Negotiation successful
0928 02:10:48.284096 (+ 374us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1093,"thread_start_us":906,"threads_started":1}
I20260928 02:10:48.291975 9040 rpc_server.cc:307] RPC server started. Bound to: 127.6.235.66:40129
I20260928 02:10:48.292007 9174 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.235.66:40129 every 8 connection(s)
I20260928 02:10:48.293416 9040 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790561432251112-7085-0/minicluster-data/ts-1/data/info.pb
I20260928 02:10:48.295655 9175 heartbeater.cc:347] Connected to a master server at 127.6.235.126:40759
I20260928 02:10:48.295921 9175 heartbeater.cc:464] Registering TS with master...
I20260928 02:10:48.296677 9175 heartbeater.cc:511] Master 127.6.235.126:40759 requested a full tablet report, sending...
I20260928 02:10:48.298139 8829 ts_manager.cc:194] Registered new tserver with Master: 97e1a30f27ab4d1480891ee30a895fad (127.6.235.66:40129)
I20260928 02:10:48.298897 8829 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.235.66@KRBTEST.COM'} at 127.6.235.66:42269
I20260928 02:10:48.300830 7085 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTlaUuc/build/asan/bin/kudu as pid 9040
I20260928 02:10:48.301043 7085 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790561432251112-7085-0/minicluster-data/ts-1/wal/instance
I20260928 02:10:48.301481 7085 external_mini_cluster.cc:1428] Setting key c80497a2fc49a1a29672c52e1c98db90
WARNING: no policy specified for kudu/127.6.235.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.235.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.235.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790561432251112-7085-0/krb5kdc/kudu_127.6.235.67.keytab.
Entry for principal kudu/127.6.235.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790561432251112-7085-0/krb5kdc/kudu_127.6.235.67.keytab.
I20260928 02:10:48.344699 7085 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTlaUuc/build/asan/bin/kudu
/tmp/dist-test-taskTlaUuc/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790561432251112-7085-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790561432251112-7085-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790561432251112-7085-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790561432251112-7085-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.235.67:0
--local_ip_for_outbound_sockets=127.6.235.67
--webserver_interface=127.6.235.67
--webserver_port=0
--tserver_master_addrs=127.6.235.126:40759
--keytab_file=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790561432251112-7085-0/krb5kdc/kudu_127.6.235.67.keytab
--principal=kudu/127.6.235.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.235.84:39741
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790561432251112-7085-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790561432251112-7085-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790561432251112-7085-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:10:48.676544 9179 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:10:48.676911 9179 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:10:48.677114 9179 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:10:48.684646 9179 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:10:48.684830 9179 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:10:48.685006 9179 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260928 02:10:48.685102 9179 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260928 02:10:48.685240 9179 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.235.67
I20260928 02:10:48.696096 9179 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.235.84:39741
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790561432251112-7085-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790561432251112-7085-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790561432251112-7085-0/krb5kdc/kudu_127.6.235.67.keytab
--principal=kudu/127.6.235.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.235.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790561432251112-7085-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.6.235.67
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.6.235.126:40759
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.235.67
--log_dir=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790561432251112-7085-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 00:51:59 UTC on ccycloud.quasar-axgkwn.root.comops.site
build id 4676464
ASAN enabled
I20260928 02:10:48.697548 9179 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:10:48.698912 9179 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:10:48.712677 9184 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:10:48.712667 9185 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:10:48.712757 9187 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:10:48.713440 9179 server_base.cc:1034] running on GCE node
I20260928 02:10:48.714098 9179 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:10:48.715052 9179 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:10:48.716261 9179 hybrid_clock.cc:648] HybridClock initialized: now 1790561448716205 us; error 49 us; skew 500 ppm
I20260928 02:10:48.716572 9179 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 02:10:48 dist-test-slave-86gv krb5kdc[8784](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561448, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.235.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:10:48.719856 9179 init.cc:377] Logged in from keytab as kudu/127.6.235.67@KRBTEST.COM (short username kudu)
I20260928 02:10:48.721468 9179 webserver.cc:460] Webserver started at http://127.6.235.67:38783/ using document root <none> and password file <none>
I20260928 02:10:48.722139 9179 fs_manager.cc:359] Metadata directory not provided
I20260928 02:10:48.722242 9179 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:10:48.722496 9179 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:10:48.725610 9179 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790561432251112-7085-0/minicluster-data/ts-2/data/instance:
uuid: "8e4e2d7ffd0947f093e84367fdf816b3"
format_stamp: "Formatted at 2026-09-28 02:10:48 on dist-test-slave-86gv"
server_key: "c058a43251db98cd955dd4ad01d45711"
server_key_iv: "a41bbb1d7f985db8ab8b1c8feff8967d"
server_key_version: "encryptionkey@0"
I20260928 02:10:48.726378 9179 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790561432251112-7085-0/minicluster-data/ts-2/wal/instance:
uuid: "8e4e2d7ffd0947f093e84367fdf816b3"
format_stamp: "Formatted at 2026-09-28 02:10:48 on dist-test-slave-86gv"
server_key: "c058a43251db98cd955dd4ad01d45711"
server_key_iv: "a41bbb1d7f985db8ab8b1c8feff8967d"
server_key_version: "encryptionkey@0"
I20260928 02:10:48.731840 9179 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260928 02:10:48.735807 9194 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:10:48.737670 9179 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.002s
I20260928 02:10:48.737831 9179 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790561432251112-7085-0/minicluster-data/ts-2/data,/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790561432251112-7085-0/minicluster-data/ts-2/wal
uuid: "8e4e2d7ffd0947f093e84367fdf816b3"
format_stamp: "Formatted at 2026-09-28 02:10:48 on dist-test-slave-86gv"
server_key: "c058a43251db98cd955dd4ad01d45711"
server_key_iv: "a41bbb1d7f985db8ab8b1c8feff8967d"
server_key_version: "encryptionkey@0"
I20260928 02:10:48.738026 9179 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790561432251112-7085-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790561432251112-7085-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790561432251112-7085-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:10:48.752635 9179 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:10:48.757414 9179 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:10:48.757670 9179 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:10:48.760689 9179 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:10:48.760865 9179 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:10:48.761009 9179 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:10:48.761097 9179 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:10:48.773096 9036 heartbeater.cc:503] Master 127.6.235.126:40759 was elected leader, sending a full tablet report...
Sep 28 02:10:48 dist-test-slave-86gv krb5kdc[8784](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561448, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.235.67@KRBTEST.COM for kudu/127.6.235.126@KRBTEST.COM
I20260928 02:10:48.789336 9203 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:10:48.762311 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.235.126:40759 (local address 127.6.235.67:42613)
0928 02:10:48.763510 (+ 1199us) negotiation.cc:107] Waiting for socket to connect
0928 02:10:48.763552 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0928 02:10:48.765934 (+ 2382us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:10:48.766757 (+ 823us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:10:48.766771 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:10:48.767647 (+ 876us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:10:48.768679 (+ 1032us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:10:48.768710 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:10:48.770204 (+ 1494us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:10:48.770212 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:10:48.772546 (+ 2334us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:10:48.772558 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:10:48.773742 (+ 1184us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:10:48.773749 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:10:48.774054 (+ 305us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:10:48.775195 (+ 1141us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:10:48.775233 (+ 38us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:10:48.778606 (+ 3373us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:10:48.783315 (+ 4709us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:10:48.783328 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:10:48.783355 (+ 27us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:10:48.784231 (+ 876us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:10:48.784732 (+ 501us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:10:48.784744 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:10:48.784750 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:10:48.785182 (+ 432us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:10:48.785957 (+ 775us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:10:48.785969 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:10:48.786611 (+ 642us) client_negotiation.cc:769] Sending connection context
0928 02:10:48.787055 (+ 444us) client_negotiation.cc:240] Negotiation successful
0928 02:10:48.787608 (+ 553us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":901,"thread_start_us":594,"threads_started":1}
I20260928 02:10:48.790038 8924 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:10:48.761530 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.235.67:42613 (local address 127.6.235.126:40759)
0928 02:10:48.761729 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0928 02:10:48.761735 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 02:10:48.763675 (+ 1940us) server_negotiation.cc:407] Connection header received
0928 02:10:48.766239 (+ 2564us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:10:48.766245 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:10:48.766338 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:10:48.766461 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:10:48.768915 (+ 2454us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:10:48.770045 (+ 1130us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:10:48.772702 (+ 2657us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:10:48.773340 (+ 638us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:10:48.773467 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:10:48.778927 (+ 5460us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:10:48.778936 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:10:48.778942 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:10:48.778999 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:10:48.783050 (+ 4051us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:10:48.784385 (+ 1335us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:10:48.784390 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:10:48.784395 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:10:48.784580 (+ 185us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:10:48.785339 (+ 759us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:10:48.785344 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:10:48.785348 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:10:48.785684 (+ 336us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:10:48.785850 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0928 02:10:48.789183 (+ 3333us) server_negotiation.cc:299] Negotiation successful
0928 02:10:48.789512 (+ 329us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260928 02:10:48.813625 8924 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:10:48.796061 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.235.67:40249 (local address 127.6.235.126:40759)
0928 02:10:48.796287 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0928 02:10:48.796296 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 02:10:48.796973 (+ 677us) server_negotiation.cc:407] Connection header received
0928 02:10:48.797214 (+ 241us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:10:48.797218 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:10:48.797374 (+ 156us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:10:48.797469 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:10:48.798978 (+ 1509us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:10:48.799726 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:10:48.802468 (+ 2742us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:10:48.803025 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:10:48.803171 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:10:48.805797 (+ 2626us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:10:48.805803 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:10:48.805806 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:10:48.805838 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:10:48.810097 (+ 4259us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:10:48.811108 (+ 1011us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:10:48.811116 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:10:48.811123 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:10:48.811298 (+ 175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:10:48.811814 (+ 516us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:10:48.811820 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:10:48.811824 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:10:48.812224 (+ 400us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:10:48.812394 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0928 02:10:48.812848 (+ 454us) server_negotiation.cc:299] Negotiation successful
0928 02:10:48.813141 (+ 293us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260928 02:10:48.813866 9275 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:10:48.796055 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.235.126:40759 (local address 127.6.235.67:40249)
0928 02:10:48.796805 (+ 750us) negotiation.cc:107] Waiting for socket to connect
0928 02:10:48.796842 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0928 02:10:48.797049 (+ 207us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:10:48.797535 (+ 486us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:10:48.797543 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:10:48.798302 (+ 759us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:10:48.798784 (+ 482us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:10:48.798801 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:10:48.799912 (+ 1111us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:10:48.799924 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:10:48.802291 (+ 2367us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:10:48.802311 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:10:48.803264 (+ 953us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:10:48.803274 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:10:48.803520 (+ 246us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:10:48.804551 (+ 1031us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:10:48.804582 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:10:48.805587 (+ 1005us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:10:48.810266 (+ 4679us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:10:48.810273 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:10:48.810279 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:10:48.810945 (+ 666us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:10:48.811448 (+ 503us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:10:48.811455 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:10:48.811465 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:10:48.811652 (+ 187us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:10:48.812356 (+ 704us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:10:48.812362 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:10:48.812572 (+ 210us) client_negotiation.cc:769] Sending connection context
0928 02:10:48.812724 (+ 152us) client_negotiation.cc:240] Negotiation successful
0928 02:10:48.813221 (+ 497us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":548,"thread_start_us":378,"threads_started":1}
I20260928 02:10:48.817852 9179 rpc_server.cc:307] RPC server started. Bound to: 127.6.235.67:33761
I20260928 02:10:48.817903 9313 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.235.67:33761 every 8 connection(s)
I20260928 02:10:48.819321 9179 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790561432251112-7085-0/minicluster-data/ts-2/data/info.pb
I20260928 02:10:48.821830 9314 heartbeater.cc:347] Connected to a master server at 127.6.235.126:40759
I20260928 02:10:48.822053 9314 heartbeater.cc:464] Registering TS with master...
I20260928 02:10:48.822716 9314 heartbeater.cc:511] Master 127.6.235.126:40759 requested a full tablet report, sending...
I20260928 02:10:48.824054 8829 ts_manager.cc:194] Registered new tserver with Master: 8e4e2d7ffd0947f093e84367fdf816b3 (127.6.235.67:33761)
I20260928 02:10:48.824635 7085 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTlaUuc/build/asan/bin/kudu as pid 9179
I20260928 02:10:48.824797 7085 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790561432251112-7085-0/minicluster-data/ts-2/wal/instance
I20260928 02:10:48.825225 8829 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.235.67@KRBTEST.COM'} at 127.6.235.67:42613
I20260928 02:10:48.825340 7085 external_mini_cluster.cc:1428] Setting key ea728e187bf1b2e7bf77fe872bfe7d3b
I20260928 02:10:48.828298 7085 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 28 02:10:48 dist-test-slave-86gv krb5kdc[8784](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561448, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 28 02:10:48 dist-test-slave-86gv krb5kdc[8784](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561448, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.235.126@KRBTEST.COM
I20260928 02:10:48.877802 8924 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:10:48.864012 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52602 (local address 127.6.235.126:40759)
0928 02:10:48.864187 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0928 02:10:48.864192 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 02:10:48.864537 (+ 345us) server_negotiation.cc:407] Connection header received
0928 02:10:48.864666 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:10:48.864670 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:10:48.864746 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:10:48.864880 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:10:48.866263 (+ 1383us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:10:48.867012 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:10:48.868204 (+ 1192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:10:48.868651 (+ 447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:10:48.868795 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:10:48.872570 (+ 3775us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:10:48.872575 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:10:48.872579 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:10:48.872614 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:10:48.875262 (+ 2648us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:10:48.875859 (+ 597us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:10:48.875865 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:10:48.875869 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:10:48.875988 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:10:48.876359 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:10:48.876366 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:10:48.876369 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:10:48.876644 (+ 275us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:10:48.876894 (+ 250us) server_negotiation.cc:1035] Waiting for connection context
0928 02:10:48.877202 (+ 308us) server_negotiation.cc:299] Negotiation successful
0928 02:10:48.877430 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260928 02:10:48.881712 8829 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52602:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 02:10:48.888366 8829 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 02:10:48.910843 9328 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:10:48.900082 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.235.67:33761 (local address 127.0.0.1:40886)
0928 02:10:48.901057 (+ 975us) negotiation.cc:107] Waiting for socket to connect
0928 02:10:48.901102 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0928 02:10:48.901298 (+ 196us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:10:48.903042 (+ 1744us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:10:48.903051 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:10:48.903075 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:10:48.903577 (+ 502us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:10:48.903588 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:10:48.905429 (+ 1841us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:10:48.905440 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:10:48.908052 (+ 2612us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:10:48.908066 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:10:48.909773 (+ 1707us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:10:48.909778 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:10:48.909961 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:10:48.910030 (+ 69us) client_negotiation.cc:769] Sending connection context
0928 02:10:48.910198 (+ 168us) client_negotiation.cc:240] Negotiation successful
0928 02:10:48.910386 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":679,"mutex_wait_us":6,"thread_start_us":1554,"threads_started":1}
I20260928 02:10:48.911381 9330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:10:48.900787 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59670 (local address 127.6.235.66:40129)
0928 02:10:48.901845 (+ 1058us) server_negotiation.cc:206] Beginning negotiation
0928 02:10:48.901854 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 02:10:48.902260 (+ 406us) server_negotiation.cc:407] Connection header received
0928 02:10:48.902355 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:10:48.902361 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:10:48.902653 (+ 292us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:10:48.903648 (+ 995us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:10:48.903836 (+ 188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:10:48.905287 (+ 1451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:10:48.908728 (+ 3441us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:10:48.909979 (+ 1251us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:10:48.910123 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:10:48.910307 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0928 02:10:48.910942 (+ 635us) server_negotiation.cc:299] Negotiation successful
0928 02:10:48.911117 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":891,"thread_start_us":776,"threads_started":1}
I20260928 02:10:48.911554 9329 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:10:48.900329 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40886 (local address 127.6.235.67:33761)
0928 02:10:48.902362 (+ 2033us) server_negotiation.cc:206] Beginning negotiation
0928 02:10:48.902371 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 02:10:48.902405 (+ 34us) server_negotiation.cc:407] Connection header received
0928 02:10:48.902519 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:10:48.902527 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:10:48.902778 (+ 251us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:10:48.903064 (+ 286us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:10:48.903826 (+ 762us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:10:48.905283 (+ 1457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:10:48.908217 (+ 2934us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:10:48.909592 (+ 1375us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:10:48.909737 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:10:48.909965 (+ 228us) server_negotiation.cc:1035] Waiting for connection context
0928 02:10:48.911076 (+ 1111us) server_negotiation.cc:299] Negotiation successful
0928 02:10:48.911227 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1880,"thread_start_us":1693,"threads_started":1}
I20260928 02:10:48.911991 9327 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:10:48.900172 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.235.66:40129 (local address 127.0.0.1:59670)
0928 02:10:48.902005 (+ 1833us) negotiation.cc:107] Waiting for socket to connect
0928 02:10:48.902042 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0928 02:10:48.902154 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:10:48.902911 (+ 757us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:10:48.902920 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:10:48.902950 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:10:48.903470 (+ 520us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:10:48.903501 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:10:48.905494 (+ 1993us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:10:48.905508 (+ 14us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:10:48.908582 (+ 3074us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:10:48.908593 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:10:48.910190 (+ 1597us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:10:48.910196 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:10:48.910552 (+ 356us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:10:48.910590 (+ 38us) client_negotiation.cc:769] Sending connection context
0928 02:10:48.910679 (+ 89us) client_negotiation.cc:240] Negotiation successful
0928 02:10:48.910828 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1637,"mutex_wait_us":30,"thread_start_us":450,"threads_started":1}
I20260928 02:10:48.912707 9332 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:10:48.902572 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.235.65:46109 (local address 127.0.0.1:52850)
0928 02:10:48.904185 (+ 1613us) negotiation.cc:107] Waiting for socket to connect
0928 02:10:48.904201 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 02:10:48.904352 (+ 151us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:10:48.905142 (+ 790us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:10:48.905147 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:10:48.905170 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:10:48.905570 (+ 400us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:10:48.905580 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:10:48.906848 (+ 1268us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:10:48.906858 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:10:48.909124 (+ 2266us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:10:48.909138 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:10:48.911796 (+ 2658us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:10:48.911802 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:10:48.911923 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:10:48.911949 (+ 26us) client_negotiation.cc:769] Sending connection context
0928 02:10:48.912242 (+ 293us) client_negotiation.cc:240] Negotiation successful
0928 02:10:48.912368 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1502,"mutex_wait_us":5,"thread_start_us":1040,"threads_started":1}
I20260928 02:10:48.912853 9331 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:10:48.902311 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52850 (local address 127.6.235.65:46109)
0928 02:10:48.902884 (+ 573us) server_negotiation.cc:206] Beginning negotiation
0928 02:10:48.902893 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 02:10:48.904470 (+ 1577us) server_negotiation.cc:407] Connection header received
0928 02:10:48.904700 (+ 230us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:10:48.904707 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:10:48.904954 (+ 247us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:10:48.905136 (+ 182us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:10:48.905741 (+ 605us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:10:48.906687 (+ 946us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:10:48.909306 (+ 2619us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:10:48.910726 (+ 1420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:10:48.910868 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:10:48.911359 (+ 491us) server_negotiation.cc:1035] Waiting for connection context
0928 02:10:48.912399 (+ 1040us) server_negotiation.cc:299] Negotiation successful
0928 02:10:48.912572 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":361,"thread_start_us":412,"threads_started":1}
I20260928 02:10:48.915148 9244 tablet_service.cc:1467] Processing CreateTablet for tablet c037b5796e644857a7d70b1c2ce4825b (DEFAULT_TABLE table=test-table [id=a5ab471f13154492bc7ff1a6cd2096a0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:10:48.915424 9105 tablet_service.cc:1467] Processing CreateTablet for tablet c037b5796e644857a7d70b1c2ce4825b (DEFAULT_TABLE table=test-table [id=a5ab471f13154492bc7ff1a6cd2096a0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:10:48.916566 9105 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c037b5796e644857a7d70b1c2ce4825b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:10:48.916606 9244 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c037b5796e644857a7d70b1c2ce4825b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:10:48.917156 8966 tablet_service.cc:1467] Processing CreateTablet for tablet c037b5796e644857a7d70b1c2ce4825b (DEFAULT_TABLE table=test-table [id=a5ab471f13154492bc7ff1a6cd2096a0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:10:48.918401 8966 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c037b5796e644857a7d70b1c2ce4825b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:10:48.927920 9333 tablet_bootstrap.cc:492] T c037b5796e644857a7d70b1c2ce4825b P 97e1a30f27ab4d1480891ee30a895fad: Bootstrap starting.
I20260928 02:10:48.930924 9334 tablet_bootstrap.cc:492] T c037b5796e644857a7d70b1c2ce4825b P 8e4e2d7ffd0947f093e84367fdf816b3: Bootstrap starting.
I20260928 02:10:48.931564 9333 tablet_bootstrap.cc:654] T c037b5796e644857a7d70b1c2ce4825b P 97e1a30f27ab4d1480891ee30a895fad: Neither blocks nor log segments found. Creating new log.
I20260928 02:10:48.933099 9333 log.cc:824] T c037b5796e644857a7d70b1c2ce4825b P 97e1a30f27ab4d1480891ee30a895fad: Log is configured to *not* fsync() on all Append() calls
I20260928 02:10:48.934700 9334 tablet_bootstrap.cc:654] T c037b5796e644857a7d70b1c2ce4825b P 8e4e2d7ffd0947f093e84367fdf816b3: Neither blocks nor log segments found. Creating new log.
I20260928 02:10:48.936036 9334 log.cc:824] T c037b5796e644857a7d70b1c2ce4825b P 8e4e2d7ffd0947f093e84367fdf816b3: Log is configured to *not* fsync() on all Append() calls
I20260928 02:10:48.937042 9335 tablet_bootstrap.cc:492] T c037b5796e644857a7d70b1c2ce4825b P 1acdcdd55e604bc498460384fd246cb4: Bootstrap starting.
I20260928 02:10:48.937086 9333 tablet_bootstrap.cc:492] T c037b5796e644857a7d70b1c2ce4825b P 97e1a30f27ab4d1480891ee30a895fad: No bootstrap required, opened a new log
I20260928 02:10:48.937357 9333 ts_tablet_manager.cc:1397] T c037b5796e644857a7d70b1c2ce4825b P 97e1a30f27ab4d1480891ee30a895fad: Time spent bootstrapping tablet: real 0.010s user 0.006s sys 0.002s
I20260928 02:10:48.938715 9334 tablet_bootstrap.cc:492] T c037b5796e644857a7d70b1c2ce4825b P 8e4e2d7ffd0947f093e84367fdf816b3: No bootstrap required, opened a new log
I20260928 02:10:48.938975 9334 ts_tablet_manager.cc:1397] T c037b5796e644857a7d70b1c2ce4825b P 8e4e2d7ffd0947f093e84367fdf816b3: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260928 02:10:48.940557 9335 tablet_bootstrap.cc:654] T c037b5796e644857a7d70b1c2ce4825b P 1acdcdd55e604bc498460384fd246cb4: Neither blocks nor log segments found. Creating new log.
I20260928 02:10:48.941857 9335 log.cc:824] T c037b5796e644857a7d70b1c2ce4825b P 1acdcdd55e604bc498460384fd246cb4: Log is configured to *not* fsync() on all Append() calls
I20260928 02:10:48.947389 9334 raft_consensus.cc:348] T c037b5796e644857a7d70b1c2ce4825b P 8e4e2d7ffd0947f093e84367fdf816b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8e4e2d7ffd0947f093e84367fdf816b3" member_type: VOTER last_known_addr { host: "127.6.235.67" port: 33761 } } peers { permanent_uuid: "97e1a30f27ab4d1480891ee30a895fad" member_type: VOTER last_known_addr { host: "127.6.235.66" port: 40129 } } peers { permanent_uuid: "1acdcdd55e604bc498460384fd246cb4" member_type: VOTER last_known_addr { host: "127.6.235.65" port: 46109 } }
I20260928 02:10:48.947778 9334 raft_consensus.cc:374] T c037b5796e644857a7d70b1c2ce4825b P 8e4e2d7ffd0947f093e84367fdf816b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:10:48.947885 9334 raft_consensus.cc:729] T c037b5796e644857a7d70b1c2ce4825b P 8e4e2d7ffd0947f093e84367fdf816b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8e4e2d7ffd0947f093e84367fdf816b3, State: Initialized, Role: FOLLOWER
I20260928 02:10:48.948522 9334 consensus_queue.cc:260] T c037b5796e644857a7d70b1c2ce4825b P 8e4e2d7ffd0947f093e84367fdf816b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8e4e2d7ffd0947f093e84367fdf816b3" member_type: VOTER last_known_addr { host: "127.6.235.67" port: 33761 } } peers { permanent_uuid: "97e1a30f27ab4d1480891ee30a895fad" member_type: VOTER last_known_addr { host: "127.6.235.66" port: 40129 } } peers { permanent_uuid: "1acdcdd55e604bc498460384fd246cb4" member_type: VOTER last_known_addr { host: "127.6.235.65" port: 46109 } }
I20260928 02:10:48.950219 9334 ts_tablet_manager.cc:1428] T c037b5796e644857a7d70b1c2ce4825b P 8e4e2d7ffd0947f093e84367fdf816b3: Time spent starting tablet: real 0.011s user 0.000s sys 0.012s
I20260928 02:10:48.950732 9314 heartbeater.cc:503] Master 127.6.235.126:40759 was elected leader, sending a full tablet report...
I20260928 02:10:48.950587 9333 raft_consensus.cc:348] T c037b5796e644857a7d70b1c2ce4825b P 97e1a30f27ab4d1480891ee30a895fad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8e4e2d7ffd0947f093e84367fdf816b3" member_type: VOTER last_known_addr { host: "127.6.235.67" port: 33761 } } peers { permanent_uuid: "97e1a30f27ab4d1480891ee30a895fad" member_type: VOTER last_known_addr { host: "127.6.235.66" port: 40129 } } peers { permanent_uuid: "1acdcdd55e604bc498460384fd246cb4" member_type: VOTER last_known_addr { host: "127.6.235.65" port: 46109 } }
I20260928 02:10:48.951056 9333 raft_consensus.cc:374] T c037b5796e644857a7d70b1c2ce4825b P 97e1a30f27ab4d1480891ee30a895fad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:10:48.951161 9333 raft_consensus.cc:729] T c037b5796e644857a7d70b1c2ce4825b P 97e1a30f27ab4d1480891ee30a895fad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 97e1a30f27ab4d1480891ee30a895fad, State: Initialized, Role: FOLLOWER
I20260928 02:10:48.951370 9335 tablet_bootstrap.cc:492] T c037b5796e644857a7d70b1c2ce4825b P 1acdcdd55e604bc498460384fd246cb4: No bootstrap required, opened a new log
I20260928 02:10:48.951665 9335 ts_tablet_manager.cc:1397] T c037b5796e644857a7d70b1c2ce4825b P 1acdcdd55e604bc498460384fd246cb4: Time spent bootstrapping tablet: real 0.017s user 0.008s sys 0.004s
I20260928 02:10:48.951753 9333 consensus_queue.cc:260] T c037b5796e644857a7d70b1c2ce4825b P 97e1a30f27ab4d1480891ee30a895fad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8e4e2d7ffd0947f093e84367fdf816b3" member_type: VOTER last_known_addr { host: "127.6.235.67" port: 33761 } } peers { permanent_uuid: "97e1a30f27ab4d1480891ee30a895fad" member_type: VOTER last_known_addr { host: "127.6.235.66" port: 40129 } } peers { permanent_uuid: "1acdcdd55e604bc498460384fd246cb4" member_type: VOTER last_known_addr { host: "127.6.235.65" port: 46109 } }
I20260928 02:10:48.957561 9175 heartbeater.cc:503] Master 127.6.235.126:40759 was elected leader, sending a full tablet report...
I20260928 02:10:48.958345 9333 ts_tablet_manager.cc:1428] T c037b5796e644857a7d70b1c2ce4825b P 97e1a30f27ab4d1480891ee30a895fad: Time spent starting tablet: real 0.021s user 0.008s sys 0.011s
I20260928 02:10:48.963081 9335 raft_consensus.cc:348] T c037b5796e644857a7d70b1c2ce4825b P 1acdcdd55e604bc498460384fd246cb4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8e4e2d7ffd0947f093e84367fdf816b3" member_type: VOTER last_known_addr { host: "127.6.235.67" port: 33761 } } peers { permanent_uuid: "97e1a30f27ab4d1480891ee30a895fad" member_type: VOTER last_known_addr { host: "127.6.235.66" port: 40129 } } peers { permanent_uuid: "1acdcdd55e604bc498460384fd246cb4" member_type: VOTER last_known_addr { host: "127.6.235.65" port: 46109 } }
I20260928 02:10:48.963476 9335 raft_consensus.cc:374] T c037b5796e644857a7d70b1c2ce4825b P 1acdcdd55e604bc498460384fd246cb4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:10:48.963583 9335 raft_consensus.cc:729] T c037b5796e644857a7d70b1c2ce4825b P 1acdcdd55e604bc498460384fd246cb4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1acdcdd55e604bc498460384fd246cb4, State: Initialized, Role: FOLLOWER
I20260928 02:10:48.964244 9335 consensus_queue.cc:260] T c037b5796e644857a7d70b1c2ce4825b P 1acdcdd55e604bc498460384fd246cb4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8e4e2d7ffd0947f093e84367fdf816b3" member_type: VOTER last_known_addr { host: "127.6.235.67" port: 33761 } } peers { permanent_uuid: "97e1a30f27ab4d1480891ee30a895fad" member_type: VOTER last_known_addr { host: "127.6.235.66" port: 40129 } } peers { permanent_uuid: "1acdcdd55e604bc498460384fd246cb4" member_type: VOTER last_known_addr { host: "127.6.235.65" port: 46109 } }
I20260928 02:10:48.966012 9335 ts_tablet_manager.cc:1428] T c037b5796e644857a7d70b1c2ce4825b P 1acdcdd55e604bc498460384fd246cb4: Time spent starting tablet: real 0.014s user 0.015s sys 0.001s
W20260928 02:10:49.013296 9037 tablet.cc:2369] T c037b5796e644857a7d70b1c2ce4825b P 1acdcdd55e604bc498460384fd246cb4: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 02:10:49.045617 9176 tablet.cc:2369] T c037b5796e644857a7d70b1c2ce4825b P 97e1a30f27ab4d1480891ee30a895fad: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 02:10:49.071136 9315 tablet.cc:2369] T c037b5796e644857a7d70b1c2ce4825b P 8e4e2d7ffd0947f093e84367fdf816b3: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 02:10:49.228336 9339 raft_consensus.cc:482] T c037b5796e644857a7d70b1c2ce4825b P 8e4e2d7ffd0947f093e84367fdf816b3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 02:10:49.228971 9339 raft_consensus.cc:504] T c037b5796e644857a7d70b1c2ce4825b P 8e4e2d7ffd0947f093e84367fdf816b3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8e4e2d7ffd0947f093e84367fdf816b3" member_type: VOTER last_known_addr { host: "127.6.235.67" port: 33761 } } peers { permanent_uuid: "97e1a30f27ab4d1480891ee30a895fad" member_type: VOTER last_known_addr { host: "127.6.235.66" port: 40129 } } peers { permanent_uuid: "1acdcdd55e604bc498460384fd246cb4" member_type: VOTER last_known_addr { host: "127.6.235.65" port: 46109 } }
I20260928 02:10:49.231248 9339 leader_election.cc:288] T c037b5796e644857a7d70b1c2ce4825b P 8e4e2d7ffd0947f093e84367fdf816b3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 97e1a30f27ab4d1480891ee30a895fad (127.6.235.66:40129), 1acdcdd55e604bc498460384fd246cb4 (127.6.235.65:46109)
I20260928 02:10:49.238718 9203 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:10:49.231574 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.235.65:46109 (local address 127.6.235.67:51949)
0928 02:10:49.232025 (+ 451us) negotiation.cc:107] Waiting for socket to connect
0928 02:10:49.232046 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0928 02:10:49.232170 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:10:49.232718 (+ 548us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:10:49.232724 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:10:49.232751 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:10:49.232987 (+ 236us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:10:49.233043 (+ 56us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:10:49.234186 (+ 1143us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:10:49.234194 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:10:49.236294 (+ 2100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:10:49.236308 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:10:49.237875 (+ 1567us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:10:49.237881 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:10:49.238108 (+ 227us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:10:49.238153 (+ 45us) client_negotiation.cc:769] Sending connection context
0928 02:10:49.238258 (+ 105us) client_negotiation.cc:240] Negotiation successful
0928 02:10:49.238404 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":310}
I20260928 02:10:49.238729 9331 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:10:49.232212 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.235.67:51949 (local address 127.6.235.65:46109)
0928 02:10:49.232431 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0928 02:10:49.232436 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 02:10:49.232451 (+ 15us) server_negotiation.cc:407] Connection header received
0928 02:10:49.232509 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:10:49.232512 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:10:49.232577 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:10:49.232681 (+ 104us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:10:49.233174 (+ 493us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:10:49.234049 (+ 875us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:10:49.236452 (+ 2403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:10:49.237719 (+ 1267us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:10:49.237888 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:10:49.237963 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0928 02:10:49.238287 (+ 324us) server_negotiation.cc:299] Negotiation successful
0928 02:10:49.238436 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260928 02:10:49.239785 8986 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c037b5796e644857a7d70b1c2ce4825b" candidate_uuid: "8e4e2d7ffd0947f093e84367fdf816b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1acdcdd55e604bc498460384fd246cb4" is_pre_election: true
I20260928 02:10:49.240324 8986 raft_consensus.cc:2454] T c037b5796e644857a7d70b1c2ce4825b P 1acdcdd55e604bc498460384fd246cb4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8e4e2d7ffd0947f093e84367fdf816b3 in term 0.
I20260928 02:10:49.240393 9330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:10:49.231904 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.235.67:55031 (local address 127.6.235.66:40129)
0928 02:10:49.232099 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0928 02:10:49.232106 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 02:10:49.232959 (+ 853us) server_negotiation.cc:407] Connection header received
0928 02:10:49.233335 (+ 376us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:10:49.233374 (+ 39us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:10:49.233469 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:10:49.233584 (+ 115us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:10:49.234472 (+ 888us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:10:49.235570 (+ 1098us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:10:49.237746 (+ 2176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:10:49.238989 (+ 1243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:10:49.239176 (+ 187us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:10:49.239251 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0928 02:10:49.239962 (+ 711us) server_negotiation.cc:299] Negotiation successful
0928 02:10:49.240097 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260928 02:10:49.240460 9342 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:10:49.231583 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.235.66:40129 (local address 127.6.235.67:55031)
0928 02:10:49.232819 (+ 1236us) negotiation.cc:107] Waiting for socket to connect
0928 02:10:49.232842 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0928 02:10:49.233098 (+ 256us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:10:49.233803 (+ 705us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:10:49.233811 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:10:49.233845 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:10:49.234288 (+ 443us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:10:49.234305 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:10:49.235720 (+ 1415us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:10:49.235727 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:10:49.237582 (+ 1855us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:10:49.237594 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:10:49.239202 (+ 1608us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:10:49.239208 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:10:49.239773 (+ 565us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:10:49.239809 (+ 36us) client_negotiation.cc:769] Sending connection context
0928 02:10:49.239923 (+ 114us) client_negotiation.cc:240] Negotiation successful
0928 02:10:49.240063 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1062,"mutex_wait_us":136,"thread_start_us":726,"threads_started":1}
I20260928 02:10:49.241070 9197 leader_election.cc:302] T c037b5796e644857a7d70b1c2ce4825b P 8e4e2d7ffd0947f093e84367fdf816b3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1acdcdd55e604bc498460384fd246cb4, 8e4e2d7ffd0947f093e84367fdf816b3; no voters:
I20260928 02:10:49.241498 9339 raft_consensus.cc:2781] T c037b5796e644857a7d70b1c2ce4825b P 8e4e2d7ffd0947f093e84367fdf816b3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 02:10:49.241638 9339 raft_consensus.cc:482] T c037b5796e644857a7d70b1c2ce4825b P 8e4e2d7ffd0947f093e84367fdf816b3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 02:10:49.241745 9339 raft_consensus.cc:3037] T c037b5796e644857a7d70b1c2ce4825b P 8e4e2d7ffd0947f093e84367fdf816b3 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:10:49.241698 9129 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c037b5796e644857a7d70b1c2ce4825b" candidate_uuid: "8e4e2d7ffd0947f093e84367fdf816b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "97e1a30f27ab4d1480891ee30a895fad" is_pre_election: true
I20260928 02:10:49.242348 9129 raft_consensus.cc:2454] T c037b5796e644857a7d70b1c2ce4825b P 97e1a30f27ab4d1480891ee30a895fad [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8e4e2d7ffd0947f093e84367fdf816b3 in term 0.
I20260928 02:10:49.244391 9339 raft_consensus.cc:504] T c037b5796e644857a7d70b1c2ce4825b P 8e4e2d7ffd0947f093e84367fdf816b3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8e4e2d7ffd0947f093e84367fdf816b3" member_type: VOTER last_known_addr { host: "127.6.235.67" port: 33761 } } peers { permanent_uuid: "97e1a30f27ab4d1480891ee30a895fad" member_type: VOTER last_known_addr { host: "127.6.235.66" port: 40129 } } peers { permanent_uuid: "1acdcdd55e604bc498460384fd246cb4" member_type: VOTER last_known_addr { host: "127.6.235.65" port: 46109 } }
I20260928 02:10:49.245081 9339 leader_election.cc:288] T c037b5796e644857a7d70b1c2ce4825b P 8e4e2d7ffd0947f093e84367fdf816b3 [CANDIDATE]: Term 1 election: Requested vote from peers 97e1a30f27ab4d1480891ee30a895fad (127.6.235.66:40129), 1acdcdd55e604bc498460384fd246cb4 (127.6.235.65:46109)
I20260928 02:10:49.245574 9129 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c037b5796e644857a7d70b1c2ce4825b" candidate_uuid: "8e4e2d7ffd0947f093e84367fdf816b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "97e1a30f27ab4d1480891ee30a895fad"
I20260928 02:10:49.245596 8986 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c037b5796e644857a7d70b1c2ce4825b" candidate_uuid: "8e4e2d7ffd0947f093e84367fdf816b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1acdcdd55e604bc498460384fd246cb4"
I20260928 02:10:49.245833 9129 raft_consensus.cc:3037] T c037b5796e644857a7d70b1c2ce4825b P 97e1a30f27ab4d1480891ee30a895fad [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:10:49.245846 8986 raft_consensus.cc:3037] T c037b5796e644857a7d70b1c2ce4825b P 1acdcdd55e604bc498460384fd246cb4 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:10:49.248700 8986 raft_consensus.cc:2454] T c037b5796e644857a7d70b1c2ce4825b P 1acdcdd55e604bc498460384fd246cb4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8e4e2d7ffd0947f093e84367fdf816b3 in term 1.
I20260928 02:10:49.249097 9197 leader_election.cc:302] T c037b5796e644857a7d70b1c2ce4825b P 8e4e2d7ffd0947f093e84367fdf816b3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1acdcdd55e604bc498460384fd246cb4, 8e4e2d7ffd0947f093e84367fdf816b3; no voters:
I20260928 02:10:49.249244 9129 raft_consensus.cc:2454] T c037b5796e644857a7d70b1c2ce4825b P 97e1a30f27ab4d1480891ee30a895fad [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8e4e2d7ffd0947f093e84367fdf816b3 in term 1.
I20260928 02:10:49.249456 9339 raft_consensus.cc:2781] T c037b5796e644857a7d70b1c2ce4825b P 8e4e2d7ffd0947f093e84367fdf816b3 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:10:49.249971 9339 raft_consensus.cc:686] T c037b5796e644857a7d70b1c2ce4825b P 8e4e2d7ffd0947f093e84367fdf816b3 [term 1 LEADER]: Becoming Leader. State: Replica: 8e4e2d7ffd0947f093e84367fdf816b3, State: Running, Role: LEADER
I20260928 02:10:49.250939 9339 consensus_queue.cc:237] T c037b5796e644857a7d70b1c2ce4825b P 8e4e2d7ffd0947f093e84367fdf816b3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8e4e2d7ffd0947f093e84367fdf816b3" member_type: VOTER last_known_addr { host: "127.6.235.67" port: 33761 } } peers { permanent_uuid: "97e1a30f27ab4d1480891ee30a895fad" member_type: VOTER last_known_addr { host: "127.6.235.66" port: 40129 } } peers { permanent_uuid: "1acdcdd55e604bc498460384fd246cb4" member_type: VOTER last_known_addr { host: "127.6.235.65" port: 46109 } }
I20260928 02:10:49.256585 8828 catalog_manager.cc:5495] T c037b5796e644857a7d70b1c2ce4825b P 8e4e2d7ffd0947f093e84367fdf816b3 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8e4e2d7ffd0947f093e84367fdf816b3 (127.6.235.67). New cstate: current_term: 1 leader_uuid: "8e4e2d7ffd0947f093e84367fdf816b3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8e4e2d7ffd0947f093e84367fdf816b3" member_type: VOTER last_known_addr { host: "127.6.235.67" port: 33761 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "97e1a30f27ab4d1480891ee30a895fad" member_type: VOTER last_known_addr { host: "127.6.235.66" port: 40129 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1acdcdd55e604bc498460384fd246cb4" member_type: VOTER last_known_addr { host: "127.6.235.65" port: 46109 } health_report { overall_health: UNKNOWN } } }
Sep 28 02:10:49 dist-test-slave-86gv krb5kdc[8784](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561447, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.235.126@KRBTEST.COM for kudu/127.6.235.126@KRBTEST.COM
I20260928 02:10:49.321177 9350 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:10:49.295956 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.235.126:40759 (local address 127.0.0.1:52614)
0928 02:10:49.296714 (+ 758us) negotiation.cc:107] Waiting for socket to connect
0928 02:10:49.296745 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0928 02:10:49.296961 (+ 216us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:10:49.297346 (+ 385us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:10:49.297351 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:10:49.298241 (+ 890us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:10:49.298622 (+ 381us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:10:49.298636 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:10:49.300182 (+ 1546us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:10:49.300189 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:10:49.302011 (+ 1822us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:10:49.302033 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:10:49.303504 (+ 1471us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:10:49.303512 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:10:49.303734 (+ 222us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:10:49.310401 (+ 6667us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:10:49.310423 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:10:49.313183 (+ 2760us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:10:49.318076 (+ 4893us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:10:49.318091 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:10:49.318107 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:10:49.318577 (+ 470us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:10:49.319055 (+ 478us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:10:49.319060 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:10:49.319064 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:10:49.319264 (+ 200us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:10:49.320096 (+ 832us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:10:49.320103 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:10:49.320259 (+ 156us) client_negotiation.cc:769] Sending connection context
0928 02:10:49.320433 (+ 174us) client_negotiation.cc:240] Negotiation successful
0928 02:10:49.320650 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":579,"thread_start_us":387,"threads_started":1}
I20260928 02:10:49.321523 8924 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:10:49.295951 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52614 (local address 127.6.235.126:40759)
0928 02:10:49.296168 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0928 02:10:49.296176 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 02:10:49.296878 (+ 702us) server_negotiation.cc:407] Connection header received
0928 02:10:49.297095 (+ 217us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:10:49.297099 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:10:49.297159 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:10:49.297284 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:10:49.298841 (+ 1557us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:10:49.300012 (+ 1171us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:10:49.302211 (+ 2199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:10:49.303031 (+ 820us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:10:49.303161 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:10:49.313443 (+ 10282us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:10:49.313448 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:10:49.313452 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:10:49.313484 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:10:49.317879 (+ 4395us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:10:49.318720 (+ 841us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:10:49.318726 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:10:49.318730 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:10:49.318912 (+ 182us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:10:49.319465 (+ 553us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:10:49.319473 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:10:49.319479 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:10:49.319935 (+ 456us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:10:49.320121 (+ 186us) server_negotiation.cc:1035] Waiting for connection context
0928 02:10:49.320597 (+ 476us) server_negotiation.cc:299] Negotiation successful
0928 02:10:49.321073 (+ 476us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260928 02:10:49.327527 8828 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.6.235.126@KRBTEST.COM'} at 127.0.0.1:52614:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260928 02:10:49.329150 8828 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 02:10:49.337474 9105 tablet_service.cc:1467] Processing CreateTablet for tablet 68f063ddf05943cc9906fb69786d0134 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=28947fcc2fd54bd4bf185cc8be8c665e]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260928 02:10:49.337324 8966 tablet_service.cc:1467] Processing CreateTablet for tablet 68f063ddf05943cc9906fb69786d0134 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=28947fcc2fd54bd4bf185cc8be8c665e]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260928 02:10:49.338436 8966 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 68f063ddf05943cc9906fb69786d0134. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:10:49.338428 9105 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 68f063ddf05943cc9906fb69786d0134. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:10:49.342105 9244 tablet_service.cc:1467] Processing CreateTablet for tablet 68f063ddf05943cc9906fb69786d0134 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=28947fcc2fd54bd4bf185cc8be8c665e]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260928 02:10:49.343246 9244 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 68f063ddf05943cc9906fb69786d0134. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:10:49.346832 9333 tablet_bootstrap.cc:492] T 68f063ddf05943cc9906fb69786d0134 P 97e1a30f27ab4d1480891ee30a895fad: Bootstrap starting.
I20260928 02:10:49.347347 9335 tablet_bootstrap.cc:492] T 68f063ddf05943cc9906fb69786d0134 P 1acdcdd55e604bc498460384fd246cb4: Bootstrap starting.
I20260928 02:10:49.348717 9333 tablet_bootstrap.cc:654] T 68f063ddf05943cc9906fb69786d0134 P 97e1a30f27ab4d1480891ee30a895fad: Neither blocks nor log segments found. Creating new log.
I20260928 02:10:49.349256 9335 tablet_bootstrap.cc:654] T 68f063ddf05943cc9906fb69786d0134 P 1acdcdd55e604bc498460384fd246cb4: Neither blocks nor log segments found. Creating new log.
I20260928 02:10:49.353545 9334 tablet_bootstrap.cc:492] T 68f063ddf05943cc9906fb69786d0134 P 8e4e2d7ffd0947f093e84367fdf816b3: Bootstrap starting.
I20260928 02:10:49.355599 9334 tablet_bootstrap.cc:654] T 68f063ddf05943cc9906fb69786d0134 P 8e4e2d7ffd0947f093e84367fdf816b3: Neither blocks nor log segments found. Creating new log.
I20260928 02:10:49.359937 9333 tablet_bootstrap.cc:492] T 68f063ddf05943cc9906fb69786d0134 P 97e1a30f27ab4d1480891ee30a895fad: No bootstrap required, opened a new log
I20260928 02:10:49.360153 9333 ts_tablet_manager.cc:1397] T 68f063ddf05943cc9906fb69786d0134 P 97e1a30f27ab4d1480891ee30a895fad: Time spent bootstrapping tablet: real 0.013s user 0.009s sys 0.001s
I20260928 02:10:49.361236 9333 raft_consensus.cc:348] T 68f063ddf05943cc9906fb69786d0134 P 97e1a30f27ab4d1480891ee30a895fad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1acdcdd55e604bc498460384fd246cb4" member_type: VOTER last_known_addr { host: "127.6.235.65" port: 46109 } } peers { permanent_uuid: "97e1a30f27ab4d1480891ee30a895fad" member_type: VOTER last_known_addr { host: "127.6.235.66" port: 40129 } } peers { permanent_uuid: "8e4e2d7ffd0947f093e84367fdf816b3" member_type: VOTER last_known_addr { host: "127.6.235.67" port: 33761 } }
I20260928 02:10:49.361557 9333 raft_consensus.cc:374] T 68f063ddf05943cc9906fb69786d0134 P 97e1a30f27ab4d1480891ee30a895fad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:10:49.361662 9333 raft_consensus.cc:729] T 68f063ddf05943cc9906fb69786d0134 P 97e1a30f27ab4d1480891ee30a895fad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 97e1a30f27ab4d1480891ee30a895fad, State: Initialized, Role: FOLLOWER
I20260928 02:10:49.362004 9333 consensus_queue.cc:260] T 68f063ddf05943cc9906fb69786d0134 P 97e1a30f27ab4d1480891ee30a895fad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1acdcdd55e604bc498460384fd246cb4" member_type: VOTER last_known_addr { host: "127.6.235.65" port: 46109 } } peers { permanent_uuid: "97e1a30f27ab4d1480891ee30a895fad" member_type: VOTER last_known_addr { host: "127.6.235.66" port: 40129 } } peers { permanent_uuid: "8e4e2d7ffd0947f093e84367fdf816b3" member_type: VOTER last_known_addr { host: "127.6.235.67" port: 33761 } }
I20260928 02:10:49.362604 9335 tablet_bootstrap.cc:492] T 68f063ddf05943cc9906fb69786d0134 P 1acdcdd55e604bc498460384fd246cb4: No bootstrap required, opened a new log
I20260928 02:10:49.362790 9335 ts_tablet_manager.cc:1397] T 68f063ddf05943cc9906fb69786d0134 P 1acdcdd55e604bc498460384fd246cb4: Time spent bootstrapping tablet: real 0.016s user 0.006s sys 0.003s
I20260928 02:10:49.362540 9340 tablet_replica.cc:439] T 68f063ddf05943cc9906fb69786d0134 P 97e1a30f27ab4d1480891ee30a895fad: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1acdcdd55e604bc498460384fd246cb4" member_type: VOTER last_known_addr { host: "127.6.235.65" port: 46109 } } peers { permanent_uuid: "97e1a30f27ab4d1480891ee30a895fad" member_type: VOTER last_known_addr { host: "127.6.235.66" port: 40129 } } peers { permanent_uuid: "8e4e2d7ffd0947f093e84367fdf816b3" member_type: VOTER last_known_addr { host: "127.6.235.67" port: 33761 } } }
I20260928 02:10:49.362936 9340 tablet_replica.cc:442] T 68f063ddf05943cc9906fb69786d0134 P 97e1a30f27ab4d1480891ee30a895fad: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 02:10:49.363936 9333 ts_tablet_manager.cc:1428] T 68f063ddf05943cc9906fb69786d0134 P 97e1a30f27ab4d1480891ee30a895fad: Time spent starting tablet: real 0.004s user 0.003s sys 0.000s
I20260928 02:10:49.363797 9335 raft_consensus.cc:348] T 68f063ddf05943cc9906fb69786d0134 P 1acdcdd55e604bc498460384fd246cb4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1acdcdd55e604bc498460384fd246cb4" member_type: VOTER last_known_addr { host: "127.6.235.65" port: 46109 } } peers { permanent_uuid: "97e1a30f27ab4d1480891ee30a895fad" member_type: VOTER last_known_addr { host: "127.6.235.66" port: 40129 } } peers { permanent_uuid: "8e4e2d7ffd0947f093e84367fdf816b3" member_type: VOTER last_known_addr { host: "127.6.235.67" port: 33761 } }
I20260928 02:10:49.364101 9335 raft_consensus.cc:374] T 68f063ddf05943cc9906fb69786d0134 P 1acdcdd55e604bc498460384fd246cb4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:10:49.364219 9335 raft_consensus.cc:729] T 68f063ddf05943cc9906fb69786d0134 P 1acdcdd55e604bc498460384fd246cb4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1acdcdd55e604bc498460384fd246cb4, State: Initialized, Role: FOLLOWER
I20260928 02:10:49.365489 9335 consensus_queue.cc:260] T 68f063ddf05943cc9906fb69786d0134 P 1acdcdd55e604bc498460384fd246cb4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1acdcdd55e604bc498460384fd246cb4" member_type: VOTER last_known_addr { host: "127.6.235.65" port: 46109 } } peers { permanent_uuid: "97e1a30f27ab4d1480891ee30a895fad" member_type: VOTER last_known_addr { host: "127.6.235.66" port: 40129 } } peers { permanent_uuid: "8e4e2d7ffd0947f093e84367fdf816b3" member_type: VOTER last_known_addr { host: "127.6.235.67" port: 33761 } }
I20260928 02:10:49.366252 9335 ts_tablet_manager.cc:1428] T 68f063ddf05943cc9906fb69786d0134 P 1acdcdd55e604bc498460384fd246cb4: Time spent starting tablet: real 0.003s user 0.002s sys 0.001s
I20260928 02:10:49.366115 9341 tablet_replica.cc:439] T 68f063ddf05943cc9906fb69786d0134 P 1acdcdd55e604bc498460384fd246cb4: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1acdcdd55e604bc498460384fd246cb4" member_type: VOTER last_known_addr { host: "127.6.235.65" port: 46109 } } peers { permanent_uuid: "97e1a30f27ab4d1480891ee30a895fad" member_type: VOTER last_known_addr { host: "127.6.235.66" port: 40129 } } peers { permanent_uuid: "8e4e2d7ffd0947f093e84367fdf816b3" member_type: VOTER last_known_addr { host: "127.6.235.67" port: 33761 } } }
I20260928 02:10:49.366547 9341 tablet_replica.cc:442] T 68f063ddf05943cc9906fb69786d0134 P 1acdcdd55e604bc498460384fd246cb4: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 02:10:49.366534 9340 raft_consensus.cc:482] T 68f063ddf05943cc9906fb69786d0134 P 97e1a30f27ab4d1480891ee30a895fad [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 02:10:49.366757 9340 raft_consensus.cc:504] T 68f063ddf05943cc9906fb69786d0134 P 97e1a30f27ab4d1480891ee30a895fad [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1acdcdd55e604bc498460384fd246cb4" member_type: VOTER last_known_addr { host: "127.6.235.65" port: 46109 } } peers { permanent_uuid: "97e1a30f27ab4d1480891ee30a895fad" member_type: VOTER last_known_addr { host: "127.6.235.66" port: 40129 } } peers { permanent_uuid: "8e4e2d7ffd0947f093e84367fdf816b3" member_type: VOTER last_known_addr { host: "127.6.235.67" port: 33761 } }
I20260928 02:10:49.367304 9334 tablet_bootstrap.cc:492] T 68f063ddf05943cc9906fb69786d0134 P 8e4e2d7ffd0947f093e84367fdf816b3: No bootstrap required, opened a new log
I20260928 02:10:49.367467 9334 ts_tablet_manager.cc:1397] T 68f063ddf05943cc9906fb69786d0134 P 8e4e2d7ffd0947f093e84367fdf816b3: Time spent bootstrapping tablet: real 0.014s user 0.006s sys 0.004s
I20260928 02:10:49.368369 9334 raft_consensus.cc:348] T 68f063ddf05943cc9906fb69786d0134 P 8e4e2d7ffd0947f093e84367fdf816b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1acdcdd55e604bc498460384fd246cb4" member_type: VOTER last_known_addr { host: "127.6.235.65" port: 46109 } } peers { permanent_uuid: "97e1a30f27ab4d1480891ee30a895fad" member_type: VOTER last_known_addr { host: "127.6.235.66" port: 40129 } } peers { permanent_uuid: "8e4e2d7ffd0947f093e84367fdf816b3" member_type: VOTER last_known_addr { host: "127.6.235.67" port: 33761 } }
I20260928 02:10:49.368534 9340 leader_election.cc:288] T 68f063ddf05943cc9906fb69786d0134 P 97e1a30f27ab4d1480891ee30a895fad [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1acdcdd55e604bc498460384fd246cb4 (127.6.235.65:46109), 8e4e2d7ffd0947f093e84367fdf816b3 (127.6.235.67:33761)
I20260928 02:10:49.368625 9334 raft_consensus.cc:374] T 68f063ddf05943cc9906fb69786d0134 P 8e4e2d7ffd0947f093e84367fdf816b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:10:49.368717 9334 raft_consensus.cc:729] T 68f063ddf05943cc9906fb69786d0134 P 8e4e2d7ffd0947f093e84367fdf816b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8e4e2d7ffd0947f093e84367fdf816b3, State: Initialized, Role: FOLLOWER
I20260928 02:10:49.369040 9334 consensus_queue.cc:260] T 68f063ddf05943cc9906fb69786d0134 P 8e4e2d7ffd0947f093e84367fdf816b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1acdcdd55e604bc498460384fd246cb4" member_type: VOTER last_known_addr { host: "127.6.235.65" port: 46109 } } peers { permanent_uuid: "97e1a30f27ab4d1480891ee30a895fad" member_type: VOTER last_known_addr { host: "127.6.235.66" port: 40129 } } peers { permanent_uuid: "8e4e2d7ffd0947f093e84367fdf816b3" member_type: VOTER last_known_addr { host: "127.6.235.67" port: 33761 } }
I20260928 02:10:49.369499 9339 tablet_replica.cc:439] T 68f063ddf05943cc9906fb69786d0134 P 8e4e2d7ffd0947f093e84367fdf816b3: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1acdcdd55e604bc498460384fd246cb4" member_type: VOTER last_known_addr { host: "127.6.235.65" port: 46109 } } peers { permanent_uuid: "97e1a30f27ab4d1480891ee30a895fad" member_type: VOTER last_known_addr { host: "127.6.235.66" port: 40129 } } peers { permanent_uuid: "8e4e2d7ffd0947f093e84367fdf816b3" member_type: VOTER last_known_addr { host: "127.6.235.67" port: 33761 } } }
I20260928 02:10:49.369797 9339 tablet_replica.cc:442] T 68f063ddf05943cc9906fb69786d0134 P 8e4e2d7ffd0947f093e84367fdf816b3: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 02:10:49.371043 9334 ts_tablet_manager.cc:1428] T 68f063ddf05943cc9906fb69786d0134 P 8e4e2d7ffd0947f093e84367fdf816b3: Time spent starting tablet: real 0.003s user 0.001s sys 0.002s
I20260928 02:10:49.382143 9331 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:10:49.369388 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.235.66:53811 (local address 127.6.235.65:46109)
0928 02:10:49.369575 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0928 02:10:49.369582 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 02:10:49.371864 (+ 2282us) server_negotiation.cc:407] Connection header received
0928 02:10:49.372154 (+ 290us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:10:49.372160 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:10:49.372249 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:10:49.372319 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:10:49.373305 (+ 986us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:10:49.374670 (+ 1365us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:10:49.378681 (+ 4011us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:10:49.380383 (+ 1702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:10:49.380549 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:10:49.380665 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0928 02:10:49.381531 (+ 866us) server_negotiation.cc:299] Negotiation successful
0928 02:10:49.381714 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260928 02:10:49.383034 9354 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:10:49.368764 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.235.65:46109 (local address 127.6.235.66:53811)
0928 02:10:49.371738 (+ 2974us) negotiation.cc:107] Waiting for socket to connect
0928 02:10:49.371763 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0928 02:10:49.371990 (+ 227us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:10:49.372545 (+ 555us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:10:49.372552 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:10:49.372583 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:10:49.373104 (+ 521us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:10:49.373121 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:10:49.374925 (+ 1804us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:10:49.374933 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:10:49.378464 (+ 3531us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:10:49.378487 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:10:49.381025 (+ 2538us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:10:49.381040 (+ 15us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:10:49.381313 (+ 273us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:10:49.381356 (+ 43us) client_negotiation.cc:769] Sending connection context
0928 02:10:49.382354 (+ 998us) client_negotiation.cc:240] Negotiation successful
0928 02:10:49.382546 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2810,"thread_start_us":2641,"threads_started":1}
I20260928 02:10:49.383832 8986 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "68f063ddf05943cc9906fb69786d0134" candidate_uuid: "97e1a30f27ab4d1480891ee30a895fad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1acdcdd55e604bc498460384fd246cb4" is_pre_election: true
I20260928 02:10:49.384052 8986 raft_consensus.cc:2454] T 68f063ddf05943cc9906fb69786d0134 P 1acdcdd55e604bc498460384fd246cb4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 97e1a30f27ab4d1480891ee30a895fad in term 0.
I20260928 02:10:49.384581 9058 leader_election.cc:302] T 68f063ddf05943cc9906fb69786d0134 P 97e1a30f27ab4d1480891ee30a895fad [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1acdcdd55e604bc498460384fd246cb4, 97e1a30f27ab4d1480891ee30a895fad; no voters:
I20260928 02:10:49.385056 9340 raft_consensus.cc:2781] T 68f063ddf05943cc9906fb69786d0134 P 97e1a30f27ab4d1480891ee30a895fad [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 02:10:49.385212 9340 raft_consensus.cc:482] T 68f063ddf05943cc9906fb69786d0134 P 97e1a30f27ab4d1480891ee30a895fad [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 02:10:49.385344 9340 raft_consensus.cc:3037] T 68f063ddf05943cc9906fb69786d0134 P 97e1a30f27ab4d1480891ee30a895fad [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:10:49.387624 9329 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:10:49.372007 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.235.66:40211 (local address 127.6.235.67:33761)
0928 02:10:49.372176 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0928 02:10:49.372182 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 02:10:49.380327 (+ 8145us) server_negotiation.cc:407] Connection header received
0928 02:10:49.380554 (+ 227us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:10:49.380560 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:10:49.380643 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:10:49.380754 (+ 111us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:10:49.381503 (+ 749us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:10:49.382326 (+ 823us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:10:49.384610 (+ 2284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:10:49.386197 (+ 1587us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:10:49.386380 (+ 183us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:10:49.386485 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0928 02:10:49.387164 (+ 679us) server_negotiation.cc:299] Negotiation successful
0928 02:10:49.387339 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260928 02:10:49.387750 9355 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:10:49.372233 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.235.67:33761 (local address 127.6.235.66:40211)
0928 02:10:49.380205 (+ 7972us) negotiation.cc:107] Waiting for socket to connect
0928 02:10:49.380222 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0928 02:10:49.380412 (+ 190us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:10:49.380773 (+ 361us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:10:49.380778 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:10:49.380807 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:10:49.381307 (+ 500us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:10:49.381317 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:10:49.382585 (+ 1268us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:10:49.382591 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:10:49.384436 (+ 1845us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:10:49.384451 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:10:49.386664 (+ 2213us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:10:49.386673 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:10:49.386942 (+ 269us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:10:49.386997 (+ 55us) client_negotiation.cc:769] Sending connection context
0928 02:10:49.387115 (+ 118us) client_negotiation.cc:240] Negotiation successful
0928 02:10:49.387303 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7823,"thread_start_us":7633,"threads_started":1}
I20260928 02:10:49.388759 9264 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "68f063ddf05943cc9906fb69786d0134" candidate_uuid: "97e1a30f27ab4d1480891ee30a895fad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8e4e2d7ffd0947f093e84367fdf816b3" is_pre_election: true
I20260928 02:10:49.388944 9340 raft_consensus.cc:504] T 68f063ddf05943cc9906fb69786d0134 P 97e1a30f27ab4d1480891ee30a895fad [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1acdcdd55e604bc498460384fd246cb4" member_type: VOTER last_known_addr { host: "127.6.235.65" port: 46109 } } peers { permanent_uuid: "97e1a30f27ab4d1480891ee30a895fad" member_type: VOTER last_known_addr { host: "127.6.235.66" port: 40129 } } peers { permanent_uuid: "8e4e2d7ffd0947f093e84367fdf816b3" member_type: VOTER last_known_addr { host: "127.6.235.67" port: 33761 } }
I20260928 02:10:49.389242 9264 raft_consensus.cc:2454] T 68f063ddf05943cc9906fb69786d0134 P 8e4e2d7ffd0947f093e84367fdf816b3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 97e1a30f27ab4d1480891ee30a895fad in term 0.
I20260928 02:10:49.389711 9340 leader_election.cc:288] T 68f063ddf05943cc9906fb69786d0134 P 97e1a30f27ab4d1480891ee30a895fad [CANDIDATE]: Term 1 election: Requested vote from peers 1acdcdd55e604bc498460384fd246cb4 (127.6.235.65:46109), 8e4e2d7ffd0947f093e84367fdf816b3 (127.6.235.67:33761)
I20260928 02:10:49.390226 8986 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "68f063ddf05943cc9906fb69786d0134" candidate_uuid: "97e1a30f27ab4d1480891ee30a895fad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1acdcdd55e604bc498460384fd246cb4"
I20260928 02:10:49.390458 8986 raft_consensus.cc:3037] T 68f063ddf05943cc9906fb69786d0134 P 1acdcdd55e604bc498460384fd246cb4 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:10:49.391136 9264 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "68f063ddf05943cc9906fb69786d0134" candidate_uuid: "97e1a30f27ab4d1480891ee30a895fad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8e4e2d7ffd0947f093e84367fdf816b3"
I20260928 02:10:49.391340 9264 raft_consensus.cc:3037] T 68f063ddf05943cc9906fb69786d0134 P 8e4e2d7ffd0947f093e84367fdf816b3 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:10:49.394062 8986 raft_consensus.cc:2454] T 68f063ddf05943cc9906fb69786d0134 P 1acdcdd55e604bc498460384fd246cb4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 97e1a30f27ab4d1480891ee30a895fad in term 1.
I20260928 02:10:49.394528 9058 leader_election.cc:302] T 68f063ddf05943cc9906fb69786d0134 P 97e1a30f27ab4d1480891ee30a895fad [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1acdcdd55e604bc498460384fd246cb4, 97e1a30f27ab4d1480891ee30a895fad; no voters:
I20260928 02:10:49.394881 9340 raft_consensus.cc:2781] T 68f063ddf05943cc9906fb69786d0134 P 97e1a30f27ab4d1480891ee30a895fad [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:10:49.395453 9340 raft_consensus.cc:686] T 68f063ddf05943cc9906fb69786d0134 P 97e1a30f27ab4d1480891ee30a895fad [term 1 LEADER]: Becoming Leader. State: Replica: 97e1a30f27ab4d1480891ee30a895fad, State: Running, Role: LEADER
I20260928 02:10:49.396216 9264 raft_consensus.cc:2454] T 68f063ddf05943cc9906fb69786d0134 P 8e4e2d7ffd0947f093e84367fdf816b3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 97e1a30f27ab4d1480891ee30a895fad in term 1.
I20260928 02:10:49.396114 9340 consensus_queue.cc:237] T 68f063ddf05943cc9906fb69786d0134 P 97e1a30f27ab4d1480891ee30a895fad [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1acdcdd55e604bc498460384fd246cb4" member_type: VOTER last_known_addr { host: "127.6.235.65" port: 46109 } } peers { permanent_uuid: "97e1a30f27ab4d1480891ee30a895fad" member_type: VOTER last_known_addr { host: "127.6.235.66" port: 40129 } } peers { permanent_uuid: "8e4e2d7ffd0947f093e84367fdf816b3" member_type: VOTER last_known_addr { host: "127.6.235.67" port: 33761 } }
I20260928 02:10:49.400306 9356 tablet_replica.cc:439] T 68f063ddf05943cc9906fb69786d0134 P 97e1a30f27ab4d1480891ee30a895fad: TxnStatusTablet state changed. Reason: New leader 97e1a30f27ab4d1480891ee30a895fad. Latest consensus state: current_term: 1 leader_uuid: "97e1a30f27ab4d1480891ee30a895fad" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1acdcdd55e604bc498460384fd246cb4" member_type: VOTER last_known_addr { host: "127.6.235.65" port: 46109 } } peers { permanent_uuid: "97e1a30f27ab4d1480891ee30a895fad" member_type: VOTER last_known_addr { host: "127.6.235.66" port: 40129 } } peers { permanent_uuid: "8e4e2d7ffd0947f093e84367fdf816b3" member_type: VOTER last_known_addr { host: "127.6.235.67" port: 33761 } } }
I20260928 02:10:49.400861 9356 tablet_replica.cc:442] T 68f063ddf05943cc9906fb69786d0134 P 97e1a30f27ab4d1480891ee30a895fad: This TxnStatusTablet replica's current role is: LEADER
I20260928 02:10:49.401957 8828 catalog_manager.cc:5495] T 68f063ddf05943cc9906fb69786d0134 P 97e1a30f27ab4d1480891ee30a895fad reported cstate change: term changed from 0 to 1, leader changed from <none> to 97e1a30f27ab4d1480891ee30a895fad (127.6.235.66). New cstate: current_term: 1 leader_uuid: "97e1a30f27ab4d1480891ee30a895fad" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1acdcdd55e604bc498460384fd246cb4" member_type: VOTER last_known_addr { host: "127.6.235.65" port: 46109 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "97e1a30f27ab4d1480891ee30a895fad" member_type: VOTER last_known_addr { host: "127.6.235.66" port: 40129 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8e4e2d7ffd0947f093e84367fdf816b3" member_type: VOTER last_known_addr { host: "127.6.235.67" port: 33761 } health_report { overall_health: UNKNOWN } } }
I20260928 02:10:49.403239 9358 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260928 02:10:49.403539 9358 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260928 02:10:49.421430 9330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:10:49.415263 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59680 (local address 127.6.235.66:40129)
0928 02:10:49.415867 (+ 604us) server_negotiation.cc:206] Beginning negotiation
0928 02:10:49.415872 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 02:10:49.415887 (+ 15us) server_negotiation.cc:407] Connection header received
0928 02:10:49.415965 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:10:49.415969 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:10:49.416039 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:10:49.416187 (+ 148us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 02:10:49.416570 (+ 383us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:10:49.417600 (+ 1030us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:10:49.418970 (+ 1370us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:10:49.419568 (+ 598us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:10:49.419725 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:10:49.420180 (+ 455us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 02:10:49.420720 (+ 540us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 02:10:49.420932 (+ 212us) server_negotiation.cc:1035] Waiting for connection context
0928 02:10:49.420988 (+ 56us) server_negotiation.cc:299] Negotiation successful
0928 02:10:49.421072 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":493}
I20260928 02:10:49.421442 9350 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:10:49.415171 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.235.66:40129 (local address 127.0.0.1:59680)
0928 02:10:49.415496 (+ 325us) negotiation.cc:107] Waiting for socket to connect
0928 02:10:49.415514 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0928 02:10:49.415631 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:10:49.416208 (+ 577us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:10:49.416213 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:10:49.416225 (+ 12us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 02:10:49.416427 (+ 202us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:10:49.416436 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:10:49.417729 (+ 1293us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:10:49.417738 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:10:49.418813 (+ 1075us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:10:49.418825 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:10:49.419740 (+ 915us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:10:49.419746 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:10:49.419914 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:10:49.419972 (+ 58us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 02:10:49.420837 (+ 865us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 02:10:49.420860 (+ 23us) client_negotiation.cc:769] Sending connection context
0928 02:10:49.420971 (+ 111us) client_negotiation.cc:240] Negotiation successful
0928 02:10:49.421125 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":153}
I20260928 02:10:49.429495 9264 raft_consensus.cc:1261] T 68f063ddf05943cc9906fb69786d0134 P 8e4e2d7ffd0947f093e84367fdf816b3 [term 1 FOLLOWER]: Refusing update from remote peer 97e1a30f27ab4d1480891ee30a895fad: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 02:10:49.430064 8986 raft_consensus.cc:1261] T 68f063ddf05943cc9906fb69786d0134 P 1acdcdd55e604bc498460384fd246cb4 [term 1 FOLLOWER]: Refusing update from remote peer 97e1a30f27ab4d1480891ee30a895fad: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 02:10:49.430543 9356 consensus_queue.cc:1035] T 68f063ddf05943cc9906fb69786d0134 P 97e1a30f27ab4d1480891ee30a895fad [LEADER]: Connected to new peer: Peer: permanent_uuid: "8e4e2d7ffd0947f093e84367fdf816b3" member_type: VOTER last_known_addr { host: "127.6.235.67" port: 33761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 02:10:49.430930 9340 consensus_queue.cc:1035] T 68f063ddf05943cc9906fb69786d0134 P 97e1a30f27ab4d1480891ee30a895fad [LEADER]: Connected to new peer: Peer: permanent_uuid: "1acdcdd55e604bc498460384fd246cb4" member_type: VOTER last_known_addr { host: "127.6.235.65" port: 46109 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 02:10:49.455561 9341 tablet_replica.cc:439] T 68f063ddf05943cc9906fb69786d0134 P 1acdcdd55e604bc498460384fd246cb4: TxnStatusTablet state changed. Reason: New leader 97e1a30f27ab4d1480891ee30a895fad. Latest consensus state: current_term: 1 leader_uuid: "97e1a30f27ab4d1480891ee30a895fad" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1acdcdd55e604bc498460384fd246cb4" member_type: VOTER last_known_addr { host: "127.6.235.65" port: 46109 } } peers { permanent_uuid: "97e1a30f27ab4d1480891ee30a895fad" member_type: VOTER last_known_addr { host: "127.6.235.66" port: 40129 } } peers { permanent_uuid: "8e4e2d7ffd0947f093e84367fdf816b3" member_type: VOTER last_known_addr { host: "127.6.235.67" port: 33761 } } }
I20260928 02:10:49.456022 9341 tablet_replica.cc:442] T 68f063ddf05943cc9906fb69786d0134 P 1acdcdd55e604bc498460384fd246cb4: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 02:10:49.459348 9361 mvcc.cc:205] Tried to move back new op lower bound from 7334139696852025344 to 7334139696731541504. Current Snapshot: MvccSnapshot[applied={T|T < 7334139696852025344}]
I20260928 02:10:49.461804 9341 tablet_replica.cc:439] T 68f063ddf05943cc9906fb69786d0134 P 1acdcdd55e604bc498460384fd246cb4: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "97e1a30f27ab4d1480891ee30a895fad" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1acdcdd55e604bc498460384fd246cb4" member_type: VOTER last_known_addr { host: "127.6.235.65" port: 46109 } } peers { permanent_uuid: "97e1a30f27ab4d1480891ee30a895fad" member_type: VOTER last_known_addr { host: "127.6.235.66" port: 40129 } } peers { permanent_uuid: "8e4e2d7ffd0947f093e84367fdf816b3" member_type: VOTER last_known_addr { host: "127.6.235.67" port: 33761 } } }
I20260928 02:10:49.462088 9341 tablet_replica.cc:442] T 68f063ddf05943cc9906fb69786d0134 P 1acdcdd55e604bc498460384fd246cb4: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 02:10:49.462611 9356 tablet_replica.cc:439] T 68f063ddf05943cc9906fb69786d0134 P 97e1a30f27ab4d1480891ee30a895fad: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "97e1a30f27ab4d1480891ee30a895fad" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1acdcdd55e604bc498460384fd246cb4" member_type: VOTER last_known_addr { host: "127.6.235.65" port: 46109 } } peers { permanent_uuid: "97e1a30f27ab4d1480891ee30a895fad" member_type: VOTER last_known_addr { host: "127.6.235.66" port: 40129 } } peers { permanent_uuid: "8e4e2d7ffd0947f093e84367fdf816b3" member_type: VOTER last_known_addr { host: "127.6.235.67" port: 33761 } } }
I20260928 02:10:49.462904 9356 tablet_replica.cc:442] T 68f063ddf05943cc9906fb69786d0134 P 97e1a30f27ab4d1480891ee30a895fad: This TxnStatusTablet replica's current role is: LEADER
I20260928 02:10:49.466605 9339 tablet_replica.cc:439] T 68f063ddf05943cc9906fb69786d0134 P 8e4e2d7ffd0947f093e84367fdf816b3: TxnStatusTablet state changed. Reason: New leader 97e1a30f27ab4d1480891ee30a895fad. Latest consensus state: current_term: 1 leader_uuid: "97e1a30f27ab4d1480891ee30a895fad" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1acdcdd55e604bc498460384fd246cb4" member_type: VOTER last_known_addr { host: "127.6.235.65" port: 46109 } } peers { permanent_uuid: "97e1a30f27ab4d1480891ee30a895fad" member_type: VOTER last_known_addr { host: "127.6.235.66" port: 40129 } } peers { permanent_uuid: "8e4e2d7ffd0947f093e84367fdf816b3" member_type: VOTER last_known_addr { host: "127.6.235.67" port: 33761 } } }
I20260928 02:10:49.466939 9339 tablet_replica.cc:442] T 68f063ddf05943cc9906fb69786d0134 P 8e4e2d7ffd0947f093e84367fdf816b3: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 02:10:49.470681 9340 tablet_replica.cc:439] T 68f063ddf05943cc9906fb69786d0134 P 97e1a30f27ab4d1480891ee30a895fad: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "97e1a30f27ab4d1480891ee30a895fad" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1acdcdd55e604bc498460384fd246cb4" member_type: VOTER last_known_addr { host: "127.6.235.65" port: 46109 } } peers { permanent_uuid: "97e1a30f27ab4d1480891ee30a895fad" member_type: VOTER last_known_addr { host: "127.6.235.66" port: 40129 } } peers { permanent_uuid: "8e4e2d7ffd0947f093e84367fdf816b3" member_type: VOTER last_known_addr { host: "127.6.235.67" port: 33761 } } }
I20260928 02:10:49.470999 9340 tablet_replica.cc:442] T 68f063ddf05943cc9906fb69786d0134 P 97e1a30f27ab4d1480891ee30a895fad: This TxnStatusTablet replica's current role is: LEADER
I20260928 02:10:49.472457 9362 mvcc.cc:205] Tried to move back new op lower bound from 7334139696852025344 to 7334139696731541504. Current Snapshot: MvccSnapshot[applied={T|T < 7334139696852025344}]
I20260928 02:10:49.479802 9363 mvcc.cc:205] Tried to move back new op lower bound from 7334139696852025344 to 7334139696731541504. Current Snapshot: MvccSnapshot[applied={T|T < 7334139696852025344}]
I20260928 02:10:49.485569 9339 tablet_replica.cc:439] T 68f063ddf05943cc9906fb69786d0134 P 8e4e2d7ffd0947f093e84367fdf816b3: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "97e1a30f27ab4d1480891ee30a895fad" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1acdcdd55e604bc498460384fd246cb4" member_type: VOTER last_known_addr { host: "127.6.235.65" port: 46109 } } peers { permanent_uuid: "97e1a30f27ab4d1480891ee30a895fad" member_type: VOTER last_known_addr { host: "127.6.235.66" port: 40129 } } peers { permanent_uuid: "8e4e2d7ffd0947f093e84367fdf816b3" member_type: VOTER last_known_addr { host: "127.6.235.67" port: 33761 } } }
I20260928 02:10:49.485912 9339 tablet_replica.cc:442] T 68f063ddf05943cc9906fb69786d0134 P 8e4e2d7ffd0947f093e84367fdf816b3: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 02:10:49.497530 9329 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:10:49.490526 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40902 (local address 127.6.235.67:33761)
0928 02:10:49.490745 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0928 02:10:49.490755 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 02:10:49.491383 (+ 628us) server_negotiation.cc:407] Connection header received
0928 02:10:49.491666 (+ 283us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:10:49.491678 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:10:49.491810 (+ 132us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:10:49.491985 (+ 175us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 02:10:49.492455 (+ 470us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:10:49.493525 (+ 1070us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:10:49.495077 (+ 1552us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:10:49.495536 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:10:49.495682 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:10:49.496189 (+ 507us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 02:10:49.496785 (+ 596us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 02:10:49.497020 (+ 235us) server_negotiation.cc:1035] Waiting for connection context
0928 02:10:49.497075 (+ 55us) server_negotiation.cc:299] Negotiation successful
0928 02:10:49.497162 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260928 02:10:49.515738 9330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:10:49.508863 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.235.67:39917 (local address 127.6.235.66:40129)
0928 02:10:49.509135 (+ 272us) server_negotiation.cc:206] Beginning negotiation
0928 02:10:49.509145 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 02:10:49.509840 (+ 695us) server_negotiation.cc:407] Connection header received
0928 02:10:49.510168 (+ 328us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:10:49.510181 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:10:49.510273 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:10:49.510399 (+ 126us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 02:10:49.510922 (+ 523us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:10:49.511820 (+ 898us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:10:49.513693 (+ 1873us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:10:49.514171 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:10:49.514329 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:10:49.514911 (+ 582us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 02:10:49.515053 (+ 142us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 02:10:49.515130 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0928 02:10:49.515384 (+ 254us) server_negotiation.cc:299] Negotiation successful
0928 02:10:49.515476 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":97}
I20260928 02:10:49.516443 9373 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:10:49.508742 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.235.66:40129 (local address 127.6.235.67:39917)
0928 02:10:49.509723 (+ 981us) negotiation.cc:107] Waiting for socket to connect
0928 02:10:49.509746 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0928 02:10:49.510003 (+ 257us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:10:49.510409 (+ 406us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:10:49.510414 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:10:49.510426 (+ 12us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 02:10:49.510760 (+ 334us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:10:49.510769 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:10:49.512025 (+ 1256us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:10:49.512032 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:10:49.513530 (+ 1498us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:10:49.513548 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:10:49.514377 (+ 829us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:10:49.514387 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:10:49.514539 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:10:49.514648 (+ 109us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 02:10:49.515218 (+ 570us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 02:10:49.515229 (+ 11us) client_negotiation.cc:769] Sending connection context
0928 02:10:49.516022 (+ 793us) client_negotiation.cc:240] Negotiation successful
0928 02:10:49.516152 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":762,"thread_start_us":627,"threads_started":1}
I20260928 02:10:49.527976 9129 raft_consensus.cc:1261] T c037b5796e644857a7d70b1c2ce4825b P 97e1a30f27ab4d1480891ee30a895fad [term 1 FOLLOWER]: Refusing update from remote peer 8e4e2d7ffd0947f093e84367fdf816b3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 02:10:49.528741 8986 raft_consensus.cc:1261] T c037b5796e644857a7d70b1c2ce4825b P 1acdcdd55e604bc498460384fd246cb4 [term 1 FOLLOWER]: Refusing update from remote peer 8e4e2d7ffd0947f093e84367fdf816b3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 02:10:49.528923 9343 consensus_queue.cc:1035] T c037b5796e644857a7d70b1c2ce4825b P 8e4e2d7ffd0947f093e84367fdf816b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "97e1a30f27ab4d1480891ee30a895fad" member_type: VOTER last_known_addr { host: "127.6.235.66" port: 40129 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 02:10:49.529596 9343 consensus_queue.cc:1035] T c037b5796e644857a7d70b1c2ce4825b P 8e4e2d7ffd0947f093e84367fdf816b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1acdcdd55e604bc498460384fd246cb4" member_type: VOTER last_known_addr { host: "127.6.235.65" port: 46109 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 02:10:49.616416 9329 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:10:49.605980 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.235.66:37945 (local address 127.6.235.67:33761)
0928 02:10:49.606265 (+ 285us) server_negotiation.cc:206] Beginning negotiation
0928 02:10:49.606274 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 02:10:49.607583 (+ 1309us) server_negotiation.cc:407] Connection header received
0928 02:10:49.607815 (+ 232us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:10:49.607821 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:10:49.607950 (+ 129us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:10:49.608055 (+ 105us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 02:10:49.609111 (+ 1056us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:10:49.610527 (+ 1416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:10:49.613555 (+ 3028us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:10:49.614322 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:10:49.614467 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:10:49.615177 (+ 710us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 02:10:49.615308 (+ 131us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 02:10:49.615391 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0928 02:10:49.615991 (+ 600us) server_negotiation.cc:299] Negotiation successful
0928 02:10:49.616084 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":95}
I20260928 02:10:49.617318 9378 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:10:49.605478 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.235.67:33761 (local address 127.6.235.66:37945)
0928 02:10:49.607442 (+ 1964us) negotiation.cc:107] Waiting for socket to connect
0928 02:10:49.607469 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0928 02:10:49.607650 (+ 181us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:10:49.608098 (+ 448us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:10:49.608105 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:10:49.608124 (+ 19us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 02:10:49.608869 (+ 745us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:10:49.608911 (+ 42us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:10:49.610743 (+ 1832us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:10:49.610752 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:10:49.613385 (+ 2633us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:10:49.613406 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:10:49.614644 (+ 1238us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:10:49.614651 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:10:49.614867 (+ 216us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:10:49.614956 (+ 89us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 02:10:49.615485 (+ 529us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 02:10:49.615495 (+ 10us) client_negotiation.cc:769] Sending connection context
0928 02:10:49.616682 (+ 1187us) client_negotiation.cc:240] Negotiation successful
0928 02:10:49.616837 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1703,"thread_start_us":1363,"threads_started":1}
I20260928 02:10:49.692198 8828 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52602:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 02:10:49.692525 8828 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52602:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 02:10:49.696946 8828 catalog_manager.cc:5782] T 00000000000000000000000000000000 P d52b1a21d6d74a8a8e0b264bdb19a5ab: Sending DeleteTablet for 3 replicas of tablet c037b5796e644857a7d70b1c2ce4825b
I20260928 02:10:49.698418 9105 tablet_service.cc:1514] Processing DeleteTablet for tablet c037b5796e644857a7d70b1c2ce4825b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:10:49 UTC) from {username='kudu', principal='kudu/127.6.235.126@KRBTEST.COM'} at 127.0.0.1:59670
I20260928 02:10:49.698436 9244 tablet_service.cc:1514] Processing DeleteTablet for tablet c037b5796e644857a7d70b1c2ce4825b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:10:49 UTC) from {username='kudu', principal='kudu/127.6.235.126@KRBTEST.COM'} at 127.0.0.1:40886
I20260928 02:10:49.699790 8966 tablet_service.cc:1514] Processing DeleteTablet for tablet c037b5796e644857a7d70b1c2ce4825b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:10:49 UTC) from {username='kudu', principal='kudu/127.6.235.126@KRBTEST.COM'} at 127.0.0.1:52850
I20260928 02:10:49.701375 8828 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52602:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 02:10:49.702060 8828 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 02:10:49.710340 9104 tablet_service.cc:1467] Processing CreateTablet for tablet 1f11c3ff59b7482b875e82a4d41b92f5 (DEFAULT_TABLE table=test-table [id=52c2c8fcef464891b8c91298d5f956ac]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:10:49.711035 9104 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1f11c3ff59b7482b875e82a4d41b92f5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:10:49.712240 9384 tablet_replica.cc:330] T c037b5796e644857a7d70b1c2ce4825b P 8e4e2d7ffd0947f093e84367fdf816b3: stopping tablet replica
I20260928 02:10:49.712828 9384 raft_consensus.cc:2229] T c037b5796e644857a7d70b1c2ce4825b P 8e4e2d7ffd0947f093e84367fdf816b3 [term 1 LEADER]: Raft consensus shutting down.
I20260928 02:10:49.714056 8965 tablet_service.cc:1467] Processing CreateTablet for tablet 1f11c3ff59b7482b875e82a4d41b92f5 (DEFAULT_TABLE table=test-table [id=52c2c8fcef464891b8c91298d5f956ac]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:10:49.714704 8965 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1f11c3ff59b7482b875e82a4d41b92f5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:10:49.719553 9385 tablet_replica.cc:330] T c037b5796e644857a7d70b1c2ce4825b P 1acdcdd55e604bc498460384fd246cb4: stopping tablet replica
I20260928 02:10:49.719812 9333 tablet_bootstrap.cc:492] T 1f11c3ff59b7482b875e82a4d41b92f5 P 97e1a30f27ab4d1480891ee30a895fad: Bootstrap starting.
I20260928 02:10:49.720013 9385 raft_consensus.cc:2229] T c037b5796e644857a7d70b1c2ce4825b P 1acdcdd55e604bc498460384fd246cb4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 02:10:49.720360 9385 raft_consensus.cc:2258] T c037b5796e644857a7d70b1c2ce4825b P 1acdcdd55e604bc498460384fd246cb4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 02:10:49.720489 9244 tablet_service.cc:1467] Processing CreateTablet for tablet 1f11c3ff59b7482b875e82a4d41b92f5 (DEFAULT_TABLE table=test-table [id=52c2c8fcef464891b8c91298d5f956ac]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:10:49.720978 9244 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1f11c3ff59b7482b875e82a4d41b92f5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:10:49.721908 9333 tablet_bootstrap.cc:654] T 1f11c3ff59b7482b875e82a4d41b92f5 P 97e1a30f27ab4d1480891ee30a895fad: Neither blocks nor log segments found. Creating new log.
I20260928 02:10:49.722455 9385 ts_tablet_manager.cc:1905] T c037b5796e644857a7d70b1c2ce4825b P 1acdcdd55e604bc498460384fd246cb4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 02:10:49.728278 9335 tablet_bootstrap.cc:492] T 1f11c3ff59b7482b875e82a4d41b92f5 P 1acdcdd55e604bc498460384fd246cb4: Bootstrap starting.
I20260928 02:10:49.730815 9335 tablet_bootstrap.cc:654] T 1f11c3ff59b7482b875e82a4d41b92f5 P 1acdcdd55e604bc498460384fd246cb4: Neither blocks nor log segments found. Creating new log.
I20260928 02:10:49.732262 9384 raft_consensus.cc:2258] T c037b5796e644857a7d70b1c2ce4825b P 8e4e2d7ffd0947f093e84367fdf816b3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 02:10:49.736739 9384 ts_tablet_manager.cc:1905] T c037b5796e644857a7d70b1c2ce4825b P 8e4e2d7ffd0947f093e84367fdf816b3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 02:10:49.742722 9335 tablet_bootstrap.cc:492] T 1f11c3ff59b7482b875e82a4d41b92f5 P 1acdcdd55e604bc498460384fd246cb4: No bootstrap required, opened a new log
I20260928 02:10:49.742954 9335 ts_tablet_manager.cc:1397] T 1f11c3ff59b7482b875e82a4d41b92f5 P 1acdcdd55e604bc498460384fd246cb4: Time spent bootstrapping tablet: real 0.015s user 0.005s sys 0.003s
I20260928 02:10:49.743351 9383 tablet_replica.cc:330] T c037b5796e644857a7d70b1c2ce4825b P 97e1a30f27ab4d1480891ee30a895fad: stopping tablet replica
I20260928 02:10:49.743825 9335 raft_consensus.cc:348] T 1f11c3ff59b7482b875e82a4d41b92f5 P 1acdcdd55e604bc498460384fd246cb4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1acdcdd55e604bc498460384fd246cb4" member_type: VOTER last_known_addr { host: "127.6.235.65" port: 46109 } } peers { permanent_uuid: "97e1a30f27ab4d1480891ee30a895fad" member_type: VOTER last_known_addr { host: "127.6.235.66" port: 40129 } } peers { permanent_uuid: "8e4e2d7ffd0947f093e84367fdf816b3" member_type: VOTER last_known_addr { host: "127.6.235.67" port: 33761 } }
I20260928 02:10:49.744107 9334 tablet_bootstrap.cc:492] T 1f11c3ff59b7482b875e82a4d41b92f5 P 8e4e2d7ffd0947f093e84367fdf816b3: Bootstrap starting.
I20260928 02:10:49.744105 9335 raft_consensus.cc:374] T 1f11c3ff59b7482b875e82a4d41b92f5 P 1acdcdd55e604bc498460384fd246cb4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:10:49.744191 9335 raft_consensus.cc:729] T 1f11c3ff59b7482b875e82a4d41b92f5 P 1acdcdd55e604bc498460384fd246cb4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1acdcdd55e604bc498460384fd246cb4, State: Initialized, Role: FOLLOWER
I20260928 02:10:49.744503 9335 consensus_queue.cc:260] T 1f11c3ff59b7482b875e82a4d41b92f5 P 1acdcdd55e604bc498460384fd246cb4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1acdcdd55e604bc498460384fd246cb4" member_type: VOTER last_known_addr { host: "127.6.235.65" port: 46109 } } peers { permanent_uuid: "97e1a30f27ab4d1480891ee30a895fad" member_type: VOTER last_known_addr { host: "127.6.235.66" port: 40129 } } peers { permanent_uuid: "8e4e2d7ffd0947f093e84367fdf816b3" member_type: VOTER last_known_addr { host: "127.6.235.67" port: 33761 } }
I20260928 02:10:49.745752 9333 tablet_bootstrap.cc:492] T 1f11c3ff59b7482b875e82a4d41b92f5 P 97e1a30f27ab4d1480891ee30a895fad: No bootstrap required, opened a new log
I20260928 02:10:49.745956 9333 ts_tablet_manager.cc:1397] T 1f11c3ff59b7482b875e82a4d41b92f5 P 97e1a30f27ab4d1480891ee30a895fad: Time spent bootstrapping tablet: real 0.026s user 0.001s sys 0.009s
I20260928 02:10:49.746059 9334 tablet_bootstrap.cc:654] T 1f11c3ff59b7482b875e82a4d41b92f5 P 8e4e2d7ffd0947f093e84367fdf816b3: Neither blocks nor log segments found. Creating new log.
I20260928 02:10:49.746196 9335 ts_tablet_manager.cc:1428] T 1f11c3ff59b7482b875e82a4d41b92f5 P 1acdcdd55e604bc498460384fd246cb4: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20260928 02:10:49.746907 9385 ts_tablet_manager.cc:1918] T c037b5796e644857a7d70b1c2ce4825b P 1acdcdd55e604bc498460384fd246cb4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 02:10:49.747099 9385 log.cc:1196] T c037b5796e644857a7d70b1c2ce4825b P 1acdcdd55e604bc498460384fd246cb4: Deleting WAL directory at /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790561432251112-7085-0/minicluster-data/ts-0/wal/wals/c037b5796e644857a7d70b1c2ce4825b
I20260928 02:10:49.747658 9385 ts_tablet_manager.cc:1939] T c037b5796e644857a7d70b1c2ce4825b P 1acdcdd55e604bc498460384fd246cb4: Deleting consensus metadata
I20260928 02:10:49.746971 9333 raft_consensus.cc:348] T 1f11c3ff59b7482b875e82a4d41b92f5 P 97e1a30f27ab4d1480891ee30a895fad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1acdcdd55e604bc498460384fd246cb4" member_type: VOTER last_known_addr { host: "127.6.235.65" port: 46109 } } peers { permanent_uuid: "97e1a30f27ab4d1480891ee30a895fad" member_type: VOTER last_known_addr { host: "127.6.235.66" port: 40129 } } peers { permanent_uuid: "8e4e2d7ffd0947f093e84367fdf816b3" member_type: VOTER last_known_addr { host: "127.6.235.67" port: 33761 } }
I20260928 02:10:49.748616 9333 raft_consensus.cc:374] T 1f11c3ff59b7482b875e82a4d41b92f5 P 97e1a30f27ab4d1480891ee30a895fad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:10:49.748747 9333 raft_consensus.cc:729] T 1f11c3ff59b7482b875e82a4d41b92f5 P 97e1a30f27ab4d1480891ee30a895fad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 97e1a30f27ab4d1480891ee30a895fad, State: Initialized, Role: FOLLOWER
I20260928 02:10:49.749094 9333 consensus_queue.cc:260] T 1f11c3ff59b7482b875e82a4d41b92f5 P 97e1a30f27ab4d1480891ee30a895fad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1acdcdd55e604bc498460384fd246cb4" member_type: VOTER last_known_addr { host: "127.6.235.65" port: 46109 } } peers { permanent_uuid: "97e1a30f27ab4d1480891ee30a895fad" member_type: VOTER last_known_addr { host: "127.6.235.66" port: 40129 } } peers { permanent_uuid: "8e4e2d7ffd0947f093e84367fdf816b3" member_type: VOTER last_known_addr { host: "127.6.235.67" port: 33761 } }
I20260928 02:10:49.750120 8815 catalog_manager.cc:4841] TS 1acdcdd55e604bc498460384fd246cb4 (127.6.235.65:46109): tablet c037b5796e644857a7d70b1c2ce4825b (table test-table [id=a5ab471f13154492bc7ff1a6cd2096a0]) successfully deleted
I20260928 02:10:49.751819 9333 ts_tablet_manager.cc:1428] T 1f11c3ff59b7482b875e82a4d41b92f5 P 97e1a30f27ab4d1480891ee30a895fad: Time spent starting tablet: real 0.006s user 0.002s sys 0.000s
I20260928 02:10:49.752247 9383 raft_consensus.cc:2229] T c037b5796e644857a7d70b1c2ce4825b P 97e1a30f27ab4d1480891ee30a895fad [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 02:10:49.759478 9384 ts_tablet_manager.cc:1918] T c037b5796e644857a7d70b1c2ce4825b P 8e4e2d7ffd0947f093e84367fdf816b3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 02:10:49.759931 9383 raft_consensus.cc:2258] T c037b5796e644857a7d70b1c2ce4825b P 97e1a30f27ab4d1480891ee30a895fad [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 02:10:49.762220 9383 ts_tablet_manager.cc:1905] T c037b5796e644857a7d70b1c2ce4825b P 97e1a30f27ab4d1480891ee30a895fad: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 02:10:49.762578 9384 log.cc:1196] T c037b5796e644857a7d70b1c2ce4825b P 8e4e2d7ffd0947f093e84367fdf816b3: Deleting WAL directory at /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790561432251112-7085-0/minicluster-data/ts-2/wal/wals/c037b5796e644857a7d70b1c2ce4825b
I20260928 02:10:49.763216 9384 ts_tablet_manager.cc:1939] T c037b5796e644857a7d70b1c2ce4825b P 8e4e2d7ffd0947f093e84367fdf816b3: Deleting consensus metadata
I20260928 02:10:49.764938 8815 catalog_manager.cc:4841] TS 8e4e2d7ffd0947f093e84367fdf816b3 (127.6.235.67:33761): tablet c037b5796e644857a7d70b1c2ce4825b (table test-table [id=a5ab471f13154492bc7ff1a6cd2096a0]) successfully deleted
I20260928 02:10:49.766381 9334 tablet_bootstrap.cc:492] T 1f11c3ff59b7482b875e82a4d41b92f5 P 8e4e2d7ffd0947f093e84367fdf816b3: No bootstrap required, opened a new log
I20260928 02:10:49.766552 9334 ts_tablet_manager.cc:1397] T 1f11c3ff59b7482b875e82a4d41b92f5 P 8e4e2d7ffd0947f093e84367fdf816b3: Time spent bootstrapping tablet: real 0.023s user 0.004s sys 0.008s
I20260928 02:10:49.767329 9334 raft_consensus.cc:348] T 1f11c3ff59b7482b875e82a4d41b92f5 P 8e4e2d7ffd0947f093e84367fdf816b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1acdcdd55e604bc498460384fd246cb4" member_type: VOTER last_known_addr { host: "127.6.235.65" port: 46109 } } peers { permanent_uuid: "97e1a30f27ab4d1480891ee30a895fad" member_type: VOTER last_known_addr { host: "127.6.235.66" port: 40129 } } peers { permanent_uuid: "8e4e2d7ffd0947f093e84367fdf816b3" member_type: VOTER last_known_addr { host: "127.6.235.67" port: 33761 } }
I20260928 02:10:49.767580 9334 raft_consensus.cc:374] T 1f11c3ff59b7482b875e82a4d41b92f5 P 8e4e2d7ffd0947f093e84367fdf816b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:10:49.767663 9334 raft_consensus.cc:729] T 1f11c3ff59b7482b875e82a4d41b92f5 P 8e4e2d7ffd0947f093e84367fdf816b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8e4e2d7ffd0947f093e84367fdf816b3, State: Initialized, Role: FOLLOWER
I20260928 02:10:49.767942 9334 consensus_queue.cc:260] T 1f11c3ff59b7482b875e82a4d41b92f5 P 8e4e2d7ffd0947f093e84367fdf816b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1acdcdd55e604bc498460384fd246cb4" member_type: VOTER last_known_addr { host: "127.6.235.65" port: 46109 } } peers { permanent_uuid: "97e1a30f27ab4d1480891ee30a895fad" member_type: VOTER last_known_addr { host: "127.6.235.66" port: 40129 } } peers { permanent_uuid: "8e4e2d7ffd0947f093e84367fdf816b3" member_type: VOTER last_known_addr { host: "127.6.235.67" port: 33761 } }
I20260928 02:10:49.769264 9334 ts_tablet_manager.cc:1428] T 1f11c3ff59b7482b875e82a4d41b92f5 P 8e4e2d7ffd0947f093e84367fdf816b3: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20260928 02:10:49.777715 9383 ts_tablet_manager.cc:1918] T c037b5796e644857a7d70b1c2ce4825b P 97e1a30f27ab4d1480891ee30a895fad: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 02:10:49.777980 9383 log.cc:1196] T c037b5796e644857a7d70b1c2ce4825b P 97e1a30f27ab4d1480891ee30a895fad: Deleting WAL directory at /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790561432251112-7085-0/minicluster-data/ts-1/wal/wals/c037b5796e644857a7d70b1c2ce4825b
I20260928 02:10:49.778596 9383 ts_tablet_manager.cc:1939] T c037b5796e644857a7d70b1c2ce4825b P 97e1a30f27ab4d1480891ee30a895fad: Deleting consensus metadata
I20260928 02:10:49.780493 8814 catalog_manager.cc:4841] TS 97e1a30f27ab4d1480891ee30a895fad (127.6.235.66:40129): tablet c037b5796e644857a7d70b1c2ce4825b (table test-table [id=a5ab471f13154492bc7ff1a6cd2096a0]) successfully deleted
I20260928 02:10:49.941166 9340 raft_consensus.cc:482] T 1f11c3ff59b7482b875e82a4d41b92f5 P 97e1a30f27ab4d1480891ee30a895fad [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 02:10:49.941401 9340 raft_consensus.cc:504] T 1f11c3ff59b7482b875e82a4d41b92f5 P 97e1a30f27ab4d1480891ee30a895fad [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1acdcdd55e604bc498460384fd246cb4" member_type: VOTER last_known_addr { host: "127.6.235.65" port: 46109 } } peers { permanent_uuid: "97e1a30f27ab4d1480891ee30a895fad" member_type: VOTER last_known_addr { host: "127.6.235.66" port: 40129 } } peers { permanent_uuid: "8e4e2d7ffd0947f093e84367fdf816b3" member_type: VOTER last_known_addr { host: "127.6.235.67" port: 33761 } }
I20260928 02:10:49.942202 9340 leader_election.cc:288] T 1f11c3ff59b7482b875e82a4d41b92f5 P 97e1a30f27ab4d1480891ee30a895fad [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1acdcdd55e604bc498460384fd246cb4 (127.6.235.65:46109), 8e4e2d7ffd0947f093e84367fdf816b3 (127.6.235.67:33761)
I20260928 02:10:49.942938 8985 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1f11c3ff59b7482b875e82a4d41b92f5" candidate_uuid: "97e1a30f27ab4d1480891ee30a895fad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1acdcdd55e604bc498460384fd246cb4" is_pre_election: true
I20260928 02:10:49.943038 9264 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1f11c3ff59b7482b875e82a4d41b92f5" candidate_uuid: "97e1a30f27ab4d1480891ee30a895fad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8e4e2d7ffd0947f093e84367fdf816b3" is_pre_election: true
I20260928 02:10:49.943190 8985 raft_consensus.cc:2454] T 1f11c3ff59b7482b875e82a4d41b92f5 P 1acdcdd55e604bc498460384fd246cb4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 97e1a30f27ab4d1480891ee30a895fad in term 0.
I20260928 02:10:49.943324 9264 raft_consensus.cc:2454] T 1f11c3ff59b7482b875e82a4d41b92f5 P 8e4e2d7ffd0947f093e84367fdf816b3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 97e1a30f27ab4d1480891ee30a895fad in term 0.
I20260928 02:10:49.943763 9058 leader_election.cc:302] T 1f11c3ff59b7482b875e82a4d41b92f5 P 97e1a30f27ab4d1480891ee30a895fad [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1acdcdd55e604bc498460384fd246cb4, 97e1a30f27ab4d1480891ee30a895fad; no voters:
I20260928 02:10:49.944100 9340 raft_consensus.cc:2781] T 1f11c3ff59b7482b875e82a4d41b92f5 P 97e1a30f27ab4d1480891ee30a895fad [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 02:10:49.944242 9340 raft_consensus.cc:482] T 1f11c3ff59b7482b875e82a4d41b92f5 P 97e1a30f27ab4d1480891ee30a895fad [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 02:10:49.944332 9340 raft_consensus.cc:3037] T 1f11c3ff59b7482b875e82a4d41b92f5 P 97e1a30f27ab4d1480891ee30a895fad [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:10:49.947751 9340 raft_consensus.cc:504] T 1f11c3ff59b7482b875e82a4d41b92f5 P 97e1a30f27ab4d1480891ee30a895fad [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1acdcdd55e604bc498460384fd246cb4" member_type: VOTER last_known_addr { host: "127.6.235.65" port: 46109 } } peers { permanent_uuid: "97e1a30f27ab4d1480891ee30a895fad" member_type: VOTER last_known_addr { host: "127.6.235.66" port: 40129 } } peers { permanent_uuid: "8e4e2d7ffd0947f093e84367fdf816b3" member_type: VOTER last_known_addr { host: "127.6.235.67" port: 33761 } }
I20260928 02:10:49.948482 9340 leader_election.cc:288] T 1f11c3ff59b7482b875e82a4d41b92f5 P 97e1a30f27ab4d1480891ee30a895fad [CANDIDATE]: Term 1 election: Requested vote from peers 1acdcdd55e604bc498460384fd246cb4 (127.6.235.65:46109), 8e4e2d7ffd0947f093e84367fdf816b3 (127.6.235.67:33761)
I20260928 02:10:49.949040 8985 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1f11c3ff59b7482b875e82a4d41b92f5" candidate_uuid: "97e1a30f27ab4d1480891ee30a895fad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1acdcdd55e604bc498460384fd246cb4"
I20260928 02:10:49.949258 8985 raft_consensus.cc:3037] T 1f11c3ff59b7482b875e82a4d41b92f5 P 1acdcdd55e604bc498460384fd246cb4 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:10:49.949268 9264 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1f11c3ff59b7482b875e82a4d41b92f5" candidate_uuid: "97e1a30f27ab4d1480891ee30a895fad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8e4e2d7ffd0947f093e84367fdf816b3"
I20260928 02:10:49.949504 9264 raft_consensus.cc:3037] T 1f11c3ff59b7482b875e82a4d41b92f5 P 8e4e2d7ffd0947f093e84367fdf816b3 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:10:49.952270 8985 raft_consensus.cc:2454] T 1f11c3ff59b7482b875e82a4d41b92f5 P 1acdcdd55e604bc498460384fd246cb4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 97e1a30f27ab4d1480891ee30a895fad in term 1.
I20260928 02:10:49.952711 9264 raft_consensus.cc:2454] T 1f11c3ff59b7482b875e82a4d41b92f5 P 8e4e2d7ffd0947f093e84367fdf816b3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 97e1a30f27ab4d1480891ee30a895fad in term 1.
I20260928 02:10:49.952885 9058 leader_election.cc:302] T 1f11c3ff59b7482b875e82a4d41b92f5 P 97e1a30f27ab4d1480891ee30a895fad [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1acdcdd55e604bc498460384fd246cb4, 97e1a30f27ab4d1480891ee30a895fad; no voters:
I20260928 02:10:49.953398 9340 raft_consensus.cc:2781] T 1f11c3ff59b7482b875e82a4d41b92f5 P 97e1a30f27ab4d1480891ee30a895fad [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:10:49.953670 9340 raft_consensus.cc:686] T 1f11c3ff59b7482b875e82a4d41b92f5 P 97e1a30f27ab4d1480891ee30a895fad [term 1 LEADER]: Becoming Leader. State: Replica: 97e1a30f27ab4d1480891ee30a895fad, State: Running, Role: LEADER
I20260928 02:10:49.954008 9340 consensus_queue.cc:237] T 1f11c3ff59b7482b875e82a4d41b92f5 P 97e1a30f27ab4d1480891ee30a895fad [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1acdcdd55e604bc498460384fd246cb4" member_type: VOTER last_known_addr { host: "127.6.235.65" port: 46109 } } peers { permanent_uuid: "97e1a30f27ab4d1480891ee30a895fad" member_type: VOTER last_known_addr { host: "127.6.235.66" port: 40129 } } peers { permanent_uuid: "8e4e2d7ffd0947f093e84367fdf816b3" member_type: VOTER last_known_addr { host: "127.6.235.67" port: 33761 } }
I20260928 02:10:49.957162 8828 catalog_manager.cc:5495] T 1f11c3ff59b7482b875e82a4d41b92f5 P 97e1a30f27ab4d1480891ee30a895fad reported cstate change: term changed from 0 to 1, leader changed from <none> to 97e1a30f27ab4d1480891ee30a895fad (127.6.235.66). New cstate: current_term: 1 leader_uuid: "97e1a30f27ab4d1480891ee30a895fad" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1acdcdd55e604bc498460384fd246cb4" member_type: VOTER last_known_addr { host: "127.6.235.65" port: 46109 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "97e1a30f27ab4d1480891ee30a895fad" member_type: VOTER last_known_addr { host: "127.6.235.66" port: 40129 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8e4e2d7ffd0947f093e84367fdf816b3" member_type: VOTER last_known_addr { host: "127.6.235.67" port: 33761 } health_report { overall_health: UNKNOWN } } }
I20260928 02:10:50.002156 9330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:10:49.995725 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59686 (local address 127.6.235.66:40129)
0928 02:10:49.995965 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0928 02:10:49.995974 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 02:10:49.996149 (+ 175us) server_negotiation.cc:407] Connection header received
0928 02:10:49.996348 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:10:49.996355 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:10:49.996439 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:10:49.996530 (+ 91us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 02:10:49.996958 (+ 428us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:10:49.998147 (+ 1189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:10:50.000057 (+ 1910us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:10:50.000761 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:10:50.000933 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:10:50.001366 (+ 433us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 02:10:50.001516 (+ 150us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 02:10:50.001608 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0928 02:10:50.001754 (+ 146us) server_negotiation.cc:299] Negotiation successful
0928 02:10:50.001845 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260928 02:10:50.017765 9378 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:10:50.011520 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.235.66:40129 (local address 127.6.235.66:33997)
0928 02:10:50.011716 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0928 02:10:50.011734 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0928 02:10:50.011844 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:10:50.012185 (+ 341us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:10:50.012190 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:10:50.012204 (+ 14us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 02:10:50.012388 (+ 184us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:10:50.012403 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:10:50.013854 (+ 1451us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:10:50.013860 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:10:50.015673 (+ 1813us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:10:50.015690 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:10:50.016590 (+ 900us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:10:50.016596 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:10:50.016768 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:10:50.016813 (+ 45us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 02:10:50.017284 (+ 471us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 02:10:50.017293 (+ 9us) client_negotiation.cc:769] Sending connection context
0928 02:10:50.017344 (+ 51us) client_negotiation.cc:240] Negotiation successful
0928 02:10:50.017469 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":41}
I20260928 02:10:50.018499 9330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:10:50.011708 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.235.66:33997 (local address 127.6.235.66:40129)
0928 02:10:50.011904 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0928 02:10:50.011910 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 02:10:50.011928 (+ 18us) server_negotiation.cc:407] Connection header received
0928 02:10:50.011981 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:10:50.011987 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:10:50.012063 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:10:50.012153 (+ 90us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 02:10:50.012538 (+ 385us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:10:50.013711 (+ 1173us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:10:50.015826 (+ 2115us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:10:50.016436 (+ 610us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:10:50.016551 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:10:50.017019 (+ 468us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 02:10:50.017182 (+ 163us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 02:10:50.018126 (+ 944us) server_negotiation.cc:1035] Waiting for connection context
0928 02:10:50.018189 (+ 63us) server_negotiation.cc:299] Negotiation successful
0928 02:10:50.018254 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260928 02:10:50.029234 8985 raft_consensus.cc:1261] T 1f11c3ff59b7482b875e82a4d41b92f5 P 1acdcdd55e604bc498460384fd246cb4 [term 1 FOLLOWER]: Refusing update from remote peer 97e1a30f27ab4d1480891ee30a895fad: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 02:10:50.029811 9264 raft_consensus.cc:1261] T 1f11c3ff59b7482b875e82a4d41b92f5 P 8e4e2d7ffd0947f093e84367fdf816b3 [term 1 FOLLOWER]: Refusing update from remote peer 97e1a30f27ab4d1480891ee30a895fad: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 02:10:50.029796 9340 consensus_queue.cc:1035] T 1f11c3ff59b7482b875e82a4d41b92f5 P 97e1a30f27ab4d1480891ee30a895fad [LEADER]: Connected to new peer: Peer: permanent_uuid: "1acdcdd55e604bc498460384fd246cb4" member_type: VOTER last_known_addr { host: "127.6.235.65" port: 46109 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 02:10:50.030635 9340 consensus_queue.cc:1035] T 1f11c3ff59b7482b875e82a4d41b92f5 P 97e1a30f27ab4d1480891ee30a895fad [LEADER]: Connected to new peer: Peer: permanent_uuid: "8e4e2d7ffd0947f093e84367fdf816b3" member_type: VOTER last_known_addr { host: "127.6.235.67" port: 33761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260928 02:10:50.066560 9361 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260928 02:10:51.665985 8829 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52602:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 02:10:51.666275 8829 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52602:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 02:10:51.670630 8829 catalog_manager.cc:5782] T 00000000000000000000000000000000 P d52b1a21d6d74a8a8e0b264bdb19a5ab: Sending DeleteTablet for 3 replicas of tablet 1f11c3ff59b7482b875e82a4d41b92f5
I20260928 02:10:51.671628 8965 tablet_service.cc:1514] Processing DeleteTablet for tablet 1f11c3ff59b7482b875e82a4d41b92f5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:10:51 UTC) from {username='kudu', principal='kudu/127.6.235.126@KRBTEST.COM'} at 127.0.0.1:52850
I20260928 02:10:51.671748 9104 tablet_service.cc:1514] Processing DeleteTablet for tablet 1f11c3ff59b7482b875e82a4d41b92f5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:10:51 UTC) from {username='kudu', principal='kudu/127.6.235.126@KRBTEST.COM'} at 127.0.0.1:59670
I20260928 02:10:51.672048 9244 tablet_service.cc:1514] Processing DeleteTablet for tablet 1f11c3ff59b7482b875e82a4d41b92f5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:10:51 UTC) from {username='kudu', principal='kudu/127.6.235.126@KRBTEST.COM'} at 127.0.0.1:40886
I20260928 02:10:51.684248 9414 tablet_replica.cc:330] T 1f11c3ff59b7482b875e82a4d41b92f5 P 1acdcdd55e604bc498460384fd246cb4: stopping tablet replica
I20260928 02:10:51.684499 9415 tablet_replica.cc:330] T 1f11c3ff59b7482b875e82a4d41b92f5 P 97e1a30f27ab4d1480891ee30a895fad: stopping tablet replica
I20260928 02:10:51.684517 9414 raft_consensus.cc:2229] T 1f11c3ff59b7482b875e82a4d41b92f5 P 1acdcdd55e604bc498460384fd246cb4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 02:10:51.684782 9414 raft_consensus.cc:2258] T 1f11c3ff59b7482b875e82a4d41b92f5 P 1acdcdd55e604bc498460384fd246cb4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 02:10:51.684746 9415 raft_consensus.cc:2229] T 1f11c3ff59b7482b875e82a4d41b92f5 P 97e1a30f27ab4d1480891ee30a895fad [term 1 LEADER]: Raft consensus shutting down.
I20260928 02:10:51.685323 9416 tablet_replica.cc:330] T 1f11c3ff59b7482b875e82a4d41b92f5 P 8e4e2d7ffd0947f093e84367fdf816b3: stopping tablet replica
I20260928 02:10:51.685554 9416 raft_consensus.cc:2229] T 1f11c3ff59b7482b875e82a4d41b92f5 P 8e4e2d7ffd0947f093e84367fdf816b3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 02:10:51.685791 9416 raft_consensus.cc:2258] T 1f11c3ff59b7482b875e82a4d41b92f5 P 8e4e2d7ffd0947f093e84367fdf816b3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 02:10:51.685909 9414 ts_tablet_manager.cc:1905] T 1f11c3ff59b7482b875e82a4d41b92f5 P 1acdcdd55e604bc498460384fd246cb4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 02:10:51.686201 9415 raft_consensus.cc:2258] T 1f11c3ff59b7482b875e82a4d41b92f5 P 97e1a30f27ab4d1480891ee30a895fad [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 02:10:51.686798 9416 ts_tablet_manager.cc:1905] T 1f11c3ff59b7482b875e82a4d41b92f5 P 8e4e2d7ffd0947f093e84367fdf816b3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 02:10:51.687256 9415 ts_tablet_manager.cc:1905] T 1f11c3ff59b7482b875e82a4d41b92f5 P 97e1a30f27ab4d1480891ee30a895fad: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 02:10:51.708140 9414 ts_tablet_manager.cc:1918] T 1f11c3ff59b7482b875e82a4d41b92f5 P 1acdcdd55e604bc498460384fd246cb4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260928 02:10:51.708318 9414 log.cc:1196] T 1f11c3ff59b7482b875e82a4d41b92f5 P 1acdcdd55e604bc498460384fd246cb4: Deleting WAL directory at /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790561432251112-7085-0/minicluster-data/ts-0/wal/wals/1f11c3ff59b7482b875e82a4d41b92f5
I20260928 02:10:51.708793 9414 ts_tablet_manager.cc:1939] T 1f11c3ff59b7482b875e82a4d41b92f5 P 1acdcdd55e604bc498460384fd246cb4: Deleting consensus metadata
Sep 28 02:10:51 dist-test-slave-86gv krb5kdc[8784](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561451, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
I20260928 02:10:51.787887 8815 catalog_manager.cc:4841] TS 1acdcdd55e604bc498460384fd246cb4 (127.6.235.65:46109): tablet 1f11c3ff59b7482b875e82a4d41b92f5 (table test-table [id=52c2c8fcef464891b8c91298d5f956ac]) successfully deleted
I20260928 02:10:51.803894 9416 ts_tablet_manager.cc:1918] T 1f11c3ff59b7482b875e82a4d41b92f5 P 8e4e2d7ffd0947f093e84367fdf816b3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260928 02:10:51.803894 9415 ts_tablet_manager.cc:1918] T 1f11c3ff59b7482b875e82a4d41b92f5 P 97e1a30f27ab4d1480891ee30a895fad: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
W20260928 02:10:51.803907 7085 mini_kdc.cc:317] Time spent kinit for test-user: real 0.131s user 0.006s sys 0.007s
I20260928 02:10:51.804082 9415 log.cc:1196] T 1f11c3ff59b7482b875e82a4d41b92f5 P 97e1a30f27ab4d1480891ee30a895fad: Deleting WAL directory at /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790561432251112-7085-0/minicluster-data/ts-1/wal/wals/1f11c3ff59b7482b875e82a4d41b92f5
I20260928 02:10:51.804077 9416 log.cc:1196] T 1f11c3ff59b7482b875e82a4d41b92f5 P 8e4e2d7ffd0947f093e84367fdf816b3: Deleting WAL directory at /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790561432251112-7085-0/minicluster-data/ts-2/wal/wals/1f11c3ff59b7482b875e82a4d41b92f5
I20260928 02:10:51.804538 9415 ts_tablet_manager.cc:1939] T 1f11c3ff59b7482b875e82a4d41b92f5 P 97e1a30f27ab4d1480891ee30a895fad: Deleting consensus metadata
I20260928 02:10:51.804574 9416 ts_tablet_manager.cc:1939] T 1f11c3ff59b7482b875e82a4d41b92f5 P 8e4e2d7ffd0947f093e84367fdf816b3: Deleting consensus metadata
I20260928 02:10:51.805764 8814 catalog_manager.cc:4841] TS 97e1a30f27ab4d1480891ee30a895fad (127.6.235.66:40129): tablet 1f11c3ff59b7482b875e82a4d41b92f5 (table test-table [id=52c2c8fcef464891b8c91298d5f956ac]) successfully deleted
I20260928 02:10:51.805922 8815 catalog_manager.cc:4841] TS 8e4e2d7ffd0947f093e84367fdf816b3 (127.6.235.67:33761): tablet 1f11c3ff59b7482b875e82a4d41b92f5 (table test-table [id=52c2c8fcef464891b8c91298d5f956ac]) successfully deleted
Sep 28 02:10:51 dist-test-slave-86gv krb5kdc[8784](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561451, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.6.235.126@KRBTEST.COM
I20260928 02:10:51.823561 9424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:10:51.808042 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55846 (local address 127.6.235.126:40759)
0928 02:10:51.808951 (+ 909us) server_negotiation.cc:206] Beginning negotiation
0928 02:10:51.808985 (+ 34us) server_negotiation.cc:399] Waiting for connection header
0928 02:10:51.809004 (+ 19us) server_negotiation.cc:407] Connection header received
0928 02:10:51.809053 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:10:51.809056 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:10:51.809114 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:10:51.809203 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:10:51.810575 (+ 1372us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:10:51.811323 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:10:51.812703 (+ 1380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:10:51.813257 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:10:51.813354 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:10:51.817683 (+ 4329us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:10:51.817688 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:10:51.817692 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:10:51.817728 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:10:51.820562 (+ 2834us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:10:51.821618 (+ 1056us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:10:51.821626 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:10:51.821632 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:10:51.821790 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:10:51.822194 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:10:51.822198 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:10:51.822200 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:10:51.822533 (+ 333us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:10:51.822713 (+ 180us) server_negotiation.cc:1035] Waiting for connection context
0928 02:10:51.823028 (+ 315us) server_negotiation.cc:299] Negotiation successful
0928 02:10:51.823198 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":762,"thread_start_us":631,"threads_started":1}
I20260928 02:10:51.827383 8829 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:55846:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 02:10:51.828030 8829 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 02:10:51.834952 8965 tablet_service.cc:1467] Processing CreateTablet for tablet be5edf70978b428393a4d6c5b7e86425 (DEFAULT_TABLE table=test-table [id=c0c0268a5f5e406ea6e9f03a9545bc3f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:10:51.835065 9104 tablet_service.cc:1467] Processing CreateTablet for tablet be5edf70978b428393a4d6c5b7e86425 (DEFAULT_TABLE table=test-table [id=c0c0268a5f5e406ea6e9f03a9545bc3f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:10:51.835453 8965 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet be5edf70978b428393a4d6c5b7e86425. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:10:51.835366 9244 tablet_service.cc:1467] Processing CreateTablet for tablet be5edf70978b428393a4d6c5b7e86425 (DEFAULT_TABLE table=test-table [id=c0c0268a5f5e406ea6e9f03a9545bc3f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:10:51.835501 9104 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet be5edf70978b428393a4d6c5b7e86425. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:10:51.835743 9244 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet be5edf70978b428393a4d6c5b7e86425. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:10:51.841723 9425 tablet_bootstrap.cc:492] T be5edf70978b428393a4d6c5b7e86425 P 97e1a30f27ab4d1480891ee30a895fad: Bootstrap starting.
I20260928 02:10:51.842443 9427 tablet_bootstrap.cc:492] T be5edf70978b428393a4d6c5b7e86425 P 8e4e2d7ffd0947f093e84367fdf816b3: Bootstrap starting.
I20260928 02:10:51.843328 9426 tablet_bootstrap.cc:492] T be5edf70978b428393a4d6c5b7e86425 P 1acdcdd55e604bc498460384fd246cb4: Bootstrap starting.
I20260928 02:10:51.843811 9425 tablet_bootstrap.cc:654] T be5edf70978b428393a4d6c5b7e86425 P 97e1a30f27ab4d1480891ee30a895fad: Neither blocks nor log segments found. Creating new log.
I20260928 02:10:51.844553 9427 tablet_bootstrap.cc:654] T be5edf70978b428393a4d6c5b7e86425 P 8e4e2d7ffd0947f093e84367fdf816b3: Neither blocks nor log segments found. Creating new log.
I20260928 02:10:51.845194 9426 tablet_bootstrap.cc:654] T be5edf70978b428393a4d6c5b7e86425 P 1acdcdd55e604bc498460384fd246cb4: Neither blocks nor log segments found. Creating new log.
I20260928 02:10:51.846458 9425 tablet_bootstrap.cc:492] T be5edf70978b428393a4d6c5b7e86425 P 97e1a30f27ab4d1480891ee30a895fad: No bootstrap required, opened a new log
I20260928 02:10:51.846649 9425 ts_tablet_manager.cc:1397] T be5edf70978b428393a4d6c5b7e86425 P 97e1a30f27ab4d1480891ee30a895fad: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260928 02:10:51.847707 9425 raft_consensus.cc:348] T be5edf70978b428393a4d6c5b7e86425 P 97e1a30f27ab4d1480891ee30a895fad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1acdcdd55e604bc498460384fd246cb4" member_type: VOTER last_known_addr { host: "127.6.235.65" port: 46109 } } peers { permanent_uuid: "97e1a30f27ab4d1480891ee30a895fad" member_type: VOTER last_known_addr { host: "127.6.235.66" port: 40129 } } peers { permanent_uuid: "8e4e2d7ffd0947f093e84367fdf816b3" member_type: VOTER last_known_addr { host: "127.6.235.67" port: 33761 } }
I20260928 02:10:51.848062 9425 raft_consensus.cc:374] T be5edf70978b428393a4d6c5b7e86425 P 97e1a30f27ab4d1480891ee30a895fad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:10:51.848168 9425 raft_consensus.cc:729] T be5edf70978b428393a4d6c5b7e86425 P 97e1a30f27ab4d1480891ee30a895fad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 97e1a30f27ab4d1480891ee30a895fad, State: Initialized, Role: FOLLOWER
I20260928 02:10:51.848459 9425 consensus_queue.cc:260] T be5edf70978b428393a4d6c5b7e86425 P 97e1a30f27ab4d1480891ee30a895fad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1acdcdd55e604bc498460384fd246cb4" member_type: VOTER last_known_addr { host: "127.6.235.65" port: 46109 } } peers { permanent_uuid: "97e1a30f27ab4d1480891ee30a895fad" member_type: VOTER last_known_addr { host: "127.6.235.66" port: 40129 } } peers { permanent_uuid: "8e4e2d7ffd0947f093e84367fdf816b3" member_type: VOTER last_known_addr { host: "127.6.235.67" port: 33761 } }
I20260928 02:10:51.849138 9425 ts_tablet_manager.cc:1428] T be5edf70978b428393a4d6c5b7e86425 P 97e1a30f27ab4d1480891ee30a895fad: Time spent starting tablet: real 0.002s user 0.000s sys 0.003s
I20260928 02:10:51.851426 9427 tablet_bootstrap.cc:492] T be5edf70978b428393a4d6c5b7e86425 P 8e4e2d7ffd0947f093e84367fdf816b3: No bootstrap required, opened a new log
I20260928 02:10:51.851593 9427 ts_tablet_manager.cc:1397] T be5edf70978b428393a4d6c5b7e86425 P 8e4e2d7ffd0947f093e84367fdf816b3: Time spent bootstrapping tablet: real 0.009s user 0.007s sys 0.000s
I20260928 02:10:51.852005 9426 tablet_bootstrap.cc:492] T be5edf70978b428393a4d6c5b7e86425 P 1acdcdd55e604bc498460384fd246cb4: No bootstrap required, opened a new log
I20260928 02:10:51.852154 9426 ts_tablet_manager.cc:1397] T be5edf70978b428393a4d6c5b7e86425 P 1acdcdd55e604bc498460384fd246cb4: Time spent bootstrapping tablet: real 0.009s user 0.008s sys 0.000s
I20260928 02:10:51.852514 9427 raft_consensus.cc:348] T be5edf70978b428393a4d6c5b7e86425 P 8e4e2d7ffd0947f093e84367fdf816b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1acdcdd55e604bc498460384fd246cb4" member_type: VOTER last_known_addr { host: "127.6.235.65" port: 46109 } } peers { permanent_uuid: "97e1a30f27ab4d1480891ee30a895fad" member_type: VOTER last_known_addr { host: "127.6.235.66" port: 40129 } } peers { permanent_uuid: "8e4e2d7ffd0947f093e84367fdf816b3" member_type: VOTER last_known_addr { host: "127.6.235.67" port: 33761 } }
I20260928 02:10:51.852778 9427 raft_consensus.cc:374] T be5edf70978b428393a4d6c5b7e86425 P 8e4e2d7ffd0947f093e84367fdf816b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:10:51.852875 9427 raft_consensus.cc:729] T be5edf70978b428393a4d6c5b7e86425 P 8e4e2d7ffd0947f093e84367fdf816b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8e4e2d7ffd0947f093e84367fdf816b3, State: Initialized, Role: FOLLOWER
I20260928 02:10:51.852985 9426 raft_consensus.cc:348] T be5edf70978b428393a4d6c5b7e86425 P 1acdcdd55e604bc498460384fd246cb4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1acdcdd55e604bc498460384fd246cb4" member_type: VOTER last_known_addr { host: "127.6.235.65" port: 46109 } } peers { permanent_uuid: "97e1a30f27ab4d1480891ee30a895fad" member_type: VOTER last_known_addr { host: "127.6.235.66" port: 40129 } } peers { permanent_uuid: "8e4e2d7ffd0947f093e84367fdf816b3" member_type: VOTER last_known_addr { host: "127.6.235.67" port: 33761 } }
I20260928 02:10:51.853232 9426 raft_consensus.cc:374] T be5edf70978b428393a4d6c5b7e86425 P 1acdcdd55e604bc498460384fd246cb4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:10:51.853315 9426 raft_consensus.cc:729] T be5edf70978b428393a4d6c5b7e86425 P 1acdcdd55e604bc498460384fd246cb4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1acdcdd55e604bc498460384fd246cb4, State: Initialized, Role: FOLLOWER
I20260928 02:10:51.853576 9426 consensus_queue.cc:260] T be5edf70978b428393a4d6c5b7e86425 P 1acdcdd55e604bc498460384fd246cb4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1acdcdd55e604bc498460384fd246cb4" member_type: VOTER last_known_addr { host: "127.6.235.65" port: 46109 } } peers { permanent_uuid: "97e1a30f27ab4d1480891ee30a895fad" member_type: VOTER last_known_addr { host: "127.6.235.66" port: 40129 } } peers { permanent_uuid: "8e4e2d7ffd0947f093e84367fdf816b3" member_type: VOTER last_known_addr { host: "127.6.235.67" port: 33761 } }
I20260928 02:10:51.853207 9427 consensus_queue.cc:260] T be5edf70978b428393a4d6c5b7e86425 P 8e4e2d7ffd0947f093e84367fdf816b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1acdcdd55e604bc498460384fd246cb4" member_type: VOTER last_known_addr { host: "127.6.235.65" port: 46109 } } peers { permanent_uuid: "97e1a30f27ab4d1480891ee30a895fad" member_type: VOTER last_known_addr { host: "127.6.235.66" port: 40129 } } peers { permanent_uuid: "8e4e2d7ffd0947f093e84367fdf816b3" member_type: VOTER last_known_addr { host: "127.6.235.67" port: 33761 } }
I20260928 02:10:51.854570 9426 ts_tablet_manager.cc:1428] T be5edf70978b428393a4d6c5b7e86425 P 1acdcdd55e604bc498460384fd246cb4: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260928 02:10:51.855324 9427 ts_tablet_manager.cc:1428] T be5edf70978b428393a4d6c5b7e86425 P 8e4e2d7ffd0947f093e84367fdf816b3: Time spent starting tablet: real 0.004s user 0.000s sys 0.004s
I20260928 02:10:51.911123 9431 raft_consensus.cc:482] T be5edf70978b428393a4d6c5b7e86425 P 1acdcdd55e604bc498460384fd246cb4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 02:10:51.911340 9431 raft_consensus.cc:504] T be5edf70978b428393a4d6c5b7e86425 P 1acdcdd55e604bc498460384fd246cb4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1acdcdd55e604bc498460384fd246cb4" member_type: VOTER last_known_addr { host: "127.6.235.65" port: 46109 } } peers { permanent_uuid: "97e1a30f27ab4d1480891ee30a895fad" member_type: VOTER last_known_addr { host: "127.6.235.66" port: 40129 } } peers { permanent_uuid: "8e4e2d7ffd0947f093e84367fdf816b3" member_type: VOTER last_known_addr { host: "127.6.235.67" port: 33761 } }
I20260928 02:10:51.913125 9431 leader_election.cc:288] T be5edf70978b428393a4d6c5b7e86425 P 1acdcdd55e604bc498460384fd246cb4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 97e1a30f27ab4d1480891ee30a895fad (127.6.235.66:40129), 8e4e2d7ffd0947f093e84367fdf816b3 (127.6.235.67:33761)
I20260928 02:10:51.930728 9434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:10:51.913506 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.235.65:59011 (local address 127.6.235.66:40129)
0928 02:10:51.924769 (+ 11263us) server_negotiation.cc:206] Beginning negotiation
0928 02:10:51.924779 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 02:10:51.925320 (+ 541us) server_negotiation.cc:407] Connection header received
0928 02:10:51.925563 (+ 243us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:10:51.925567 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:10:51.925624 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:10:51.925736 (+ 112us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:10:51.926447 (+ 711us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:10:51.927234 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:10:51.928938 (+ 1704us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:10:51.929713 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:10:51.929912 (+ 199us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:10:51.929989 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0928 02:10:51.930246 (+ 257us) server_negotiation.cc:299] Negotiation successful
0928 02:10:51.930375 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11048,"thread_start_us":10990,"threads_started":1}
I20260928 02:10:51.930845 9435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:10:51.913280 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.235.66:40129 (local address 127.6.235.65:59011)
0928 02:10:51.925172 (+ 11892us) negotiation.cc:107] Waiting for socket to connect
0928 02:10:51.925194 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0928 02:10:51.925407 (+ 213us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:10:51.925751 (+ 344us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:10:51.925756 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:10:51.926035 (+ 279us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:10:51.926280 (+ 245us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:10:51.926289 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:10:51.927379 (+ 1090us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:10:51.927385 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:10:51.928749 (+ 1364us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:10:51.928763 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:10:51.929912 (+ 1149us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:10:51.929918 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:10:51.930083 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:10:51.930106 (+ 23us) client_negotiation.cc:769] Sending connection context
0928 02:10:51.930208 (+ 102us) client_negotiation.cc:240] Negotiation successful
0928 02:10:51.930341 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11717,"mutex_wait_us":1,"spinlock_wait_cycles":547584,"thread_start_us":11730,"threads_started":1}
I20260928 02:10:51.931463 9433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:10:51.913524 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.235.67:33761 (local address 127.6.235.65:50069)
0928 02:10:51.925438 (+ 11914us) negotiation.cc:107] Waiting for socket to connect
0928 02:10:51.925457 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0928 02:10:51.926004 (+ 547us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:10:51.926369 (+ 365us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:10:51.926373 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:10:51.926395 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:10:51.926719 (+ 324us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:10:51.926727 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:10:51.927850 (+ 1123us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:10:51.927854 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:10:51.929284 (+ 1430us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:10:51.929296 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:10:51.930550 (+ 1254us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:10:51.930556 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:10:51.930729 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:10:51.930754 (+ 25us) client_negotiation.cc:769] Sending connection context
0928 02:10:51.930874 (+ 120us) client_negotiation.cc:240] Negotiation successful
0928 02:10:51.931028 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11801,"thread_start_us":11261,"threads_started":1}
I20260928 02:10:51.931686 9436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:10:51.913660 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.235.65:50069 (local address 127.6.235.67:33761)
0928 02:10:51.924462 (+ 10802us) server_negotiation.cc:206] Beginning negotiation
0928 02:10:51.924472 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 02:10:51.925568 (+ 1096us) server_negotiation.cc:407] Connection header received
0928 02:10:51.926167 (+ 599us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:10:51.926174 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:10:51.926239 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:10:51.926356 (+ 117us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:10:51.926916 (+ 560us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:10:51.927682 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:10:51.929444 (+ 1762us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:10:51.930338 (+ 894us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:10:51.930542 (+ 204us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:10:51.930615 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0928 02:10:51.931184 (+ 569us) server_negotiation.cc:299] Negotiation successful
0928 02:10:51.931342 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10593,"thread_start_us":10405,"threads_started":1}
I20260928 02:10:51.931655 9128 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be5edf70978b428393a4d6c5b7e86425" candidate_uuid: "1acdcdd55e604bc498460384fd246cb4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "97e1a30f27ab4d1480891ee30a895fad" is_pre_election: true
I20260928 02:10:51.931918 9128 raft_consensus.cc:2454] T be5edf70978b428393a4d6c5b7e86425 P 97e1a30f27ab4d1480891ee30a895fad [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1acdcdd55e604bc498460384fd246cb4 in term 0.
I20260928 02:10:51.932288 9263 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be5edf70978b428393a4d6c5b7e86425" candidate_uuid: "1acdcdd55e604bc498460384fd246cb4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8e4e2d7ffd0947f093e84367fdf816b3" is_pre_election: true
I20260928 02:10:51.932554 9263 raft_consensus.cc:2454] T be5edf70978b428393a4d6c5b7e86425 P 8e4e2d7ffd0947f093e84367fdf816b3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1acdcdd55e604bc498460384fd246cb4 in term 0.
I20260928 02:10:51.932469 8917 leader_election.cc:302] T be5edf70978b428393a4d6c5b7e86425 P 1acdcdd55e604bc498460384fd246cb4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1acdcdd55e604bc498460384fd246cb4, 97e1a30f27ab4d1480891ee30a895fad; no voters:
I20260928 02:10:51.932933 9431 raft_consensus.cc:2781] T be5edf70978b428393a4d6c5b7e86425 P 1acdcdd55e604bc498460384fd246cb4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 02:10:51.933084 9431 raft_consensus.cc:482] T be5edf70978b428393a4d6c5b7e86425 P 1acdcdd55e604bc498460384fd246cb4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 02:10:51.933212 9431 raft_consensus.cc:3037] T be5edf70978b428393a4d6c5b7e86425 P 1acdcdd55e604bc498460384fd246cb4 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:10:52.012691 9431 raft_consensus.cc:504] T be5edf70978b428393a4d6c5b7e86425 P 1acdcdd55e604bc498460384fd246cb4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1acdcdd55e604bc498460384fd246cb4" member_type: VOTER last_known_addr { host: "127.6.235.65" port: 46109 } } peers { permanent_uuid: "97e1a30f27ab4d1480891ee30a895fad" member_type: VOTER last_known_addr { host: "127.6.235.66" port: 40129 } } peers { permanent_uuid: "8e4e2d7ffd0947f093e84367fdf816b3" member_type: VOTER last_known_addr { host: "127.6.235.67" port: 33761 } }
I20260928 02:10:52.013638 9431 leader_election.cc:288] T be5edf70978b428393a4d6c5b7e86425 P 1acdcdd55e604bc498460384fd246cb4 [CANDIDATE]: Term 1 election: Requested vote from peers 97e1a30f27ab4d1480891ee30a895fad (127.6.235.66:40129), 8e4e2d7ffd0947f093e84367fdf816b3 (127.6.235.67:33761)
I20260928 02:10:52.014333 9128 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be5edf70978b428393a4d6c5b7e86425" candidate_uuid: "1acdcdd55e604bc498460384fd246cb4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "97e1a30f27ab4d1480891ee30a895fad"
I20260928 02:10:52.014566 9128 raft_consensus.cc:3037] T be5edf70978b428393a4d6c5b7e86425 P 97e1a30f27ab4d1480891ee30a895fad [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:10:52.014461 9263 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be5edf70978b428393a4d6c5b7e86425" candidate_uuid: "1acdcdd55e604bc498460384fd246cb4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8e4e2d7ffd0947f093e84367fdf816b3"
I20260928 02:10:52.014739 9263 raft_consensus.cc:3037] T be5edf70978b428393a4d6c5b7e86425 P 8e4e2d7ffd0947f093e84367fdf816b3 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:10:52.017022 9128 raft_consensus.cc:2454] T be5edf70978b428393a4d6c5b7e86425 P 97e1a30f27ab4d1480891ee30a895fad [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1acdcdd55e604bc498460384fd246cb4 in term 1.
I20260928 02:10:52.017565 8917 leader_election.cc:302] T be5edf70978b428393a4d6c5b7e86425 P 1acdcdd55e604bc498460384fd246cb4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1acdcdd55e604bc498460384fd246cb4, 97e1a30f27ab4d1480891ee30a895fad; no voters:
I20260928 02:10:52.017897 9431 raft_consensus.cc:2781] T be5edf70978b428393a4d6c5b7e86425 P 1acdcdd55e604bc498460384fd246cb4 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:10:52.018458 9431 raft_consensus.cc:686] T be5edf70978b428393a4d6c5b7e86425 P 1acdcdd55e604bc498460384fd246cb4 [term 1 LEADER]: Becoming Leader. State: Replica: 1acdcdd55e604bc498460384fd246cb4, State: Running, Role: LEADER
I20260928 02:10:52.018893 9431 consensus_queue.cc:237] T be5edf70978b428393a4d6c5b7e86425 P 1acdcdd55e604bc498460384fd246cb4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1acdcdd55e604bc498460384fd246cb4" member_type: VOTER last_known_addr { host: "127.6.235.65" port: 46109 } } peers { permanent_uuid: "97e1a30f27ab4d1480891ee30a895fad" member_type: VOTER last_known_addr { host: "127.6.235.66" port: 40129 } } peers { permanent_uuid: "8e4e2d7ffd0947f093e84367fdf816b3" member_type: VOTER last_known_addr { host: "127.6.235.67" port: 33761 } }
I20260928 02:10:52.022272 8828 catalog_manager.cc:5495] T be5edf70978b428393a4d6c5b7e86425 P 1acdcdd55e604bc498460384fd246cb4 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1acdcdd55e604bc498460384fd246cb4 (127.6.235.65). New cstate: current_term: 1 leader_uuid: "1acdcdd55e604bc498460384fd246cb4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1acdcdd55e604bc498460384fd246cb4" member_type: VOTER last_known_addr { host: "127.6.235.65" port: 46109 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "97e1a30f27ab4d1480891ee30a895fad" member_type: VOTER last_known_addr { host: "127.6.235.66" port: 40129 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8e4e2d7ffd0947f093e84367fdf816b3" member_type: VOTER last_known_addr { host: "127.6.235.67" port: 33761 } health_report { overall_health: UNKNOWN } } }
I20260928 02:10:52.047581 9440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:10:52.041463 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41522 (local address 127.6.235.65:46109)
0928 02:10:52.042136 (+ 673us) server_negotiation.cc:206] Beginning negotiation
0928 02:10:52.042142 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 02:10:52.042162 (+ 20us) server_negotiation.cc:407] Connection header received
0928 02:10:52.042272 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:10:52.042276 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:10:52.042349 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:10:52.042529 (+ 180us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 02:10:52.042975 (+ 446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:10:52.043841 (+ 866us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:10:52.045248 (+ 1407us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:10:52.045670 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:10:52.045798 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:10:52.046267 (+ 469us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 02:10:52.046912 (+ 645us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 02:10:52.047072 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0928 02:10:52.047128 (+ 56us) server_negotiation.cc:299] Negotiation successful
0928 02:10:52.047206 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":508,"thread_start_us":300,"threads_started":1}
I20260928 02:10:52.063830 9434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:10:52.058374 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.235.65:51791 (local address 127.6.235.66:40129)
0928 02:10:52.058541 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0928 02:10:52.058545 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:10:52.058924 (+ 379us) server_negotiation.cc:407] Connection header received
0928 02:10:52.059070 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:10:52.059073 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:10:52.059136 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:10:52.059211 (+ 75us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 02:10:52.059791 (+ 580us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:10:52.060498 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:10:52.062203 (+ 1705us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:10:52.062818 (+ 615us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:10:52.062911 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:10:52.063312 (+ 401us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 02:10:52.063404 (+ 92us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 02:10:52.063500 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0928 02:10:52.063580 (+ 80us) server_negotiation.cc:299] Negotiation successful
0928 02:10:52.063638 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260928 02:10:52.063983 9443 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:10:52.058138 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.235.66:40129 (local address 127.6.235.65:51791)
0928 02:10:52.058819 (+ 681us) negotiation.cc:107] Waiting for socket to connect
0928 02:10:52.058841 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0928 02:10:52.058961 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:10:52.059270 (+ 309us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:10:52.059274 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:10:52.059284 (+ 10us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 02:10:52.059641 (+ 357us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:10:52.059656 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:10:52.060649 (+ 993us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:10:52.060653 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:10:52.062079 (+ 1426us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:10:52.062092 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:10:52.062978 (+ 886us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:10:52.062983 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:10:52.063115 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:10:52.063172 (+ 57us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 02:10:52.063484 (+ 312us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 02:10:52.063492 (+ 8us) client_negotiation.cc:769] Sending connection context
0928 02:10:52.063549 (+ 57us) client_negotiation.cc:240] Negotiation successful
0928 02:10:52.063645 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":481,"thread_start_us":351,"threads_started":1}
I20260928 02:10:52.072742 9128 raft_consensus.cc:1261] T be5edf70978b428393a4d6c5b7e86425 P 97e1a30f27ab4d1480891ee30a895fad [term 1 FOLLOWER]: Refusing update from remote peer 1acdcdd55e604bc498460384fd246cb4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 02:10:52.073540 9437 consensus_queue.cc:1035] T be5edf70978b428393a4d6c5b7e86425 P 1acdcdd55e604bc498460384fd246cb4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "97e1a30f27ab4d1480891ee30a895fad" member_type: VOTER last_known_addr { host: "127.6.235.66" port: 40129 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 02:10:52.104785 9440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:10:52.092403 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.235.66:59153 (local address 127.6.235.65:46109)
0928 02:10:52.092557 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0928 02:10:52.092563 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 02:10:52.098646 (+ 6083us) server_negotiation.cc:407] Connection header received
0928 02:10:52.098795 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:10:52.098800 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:10:52.098894 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:10:52.098988 (+ 94us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 02:10:52.099536 (+ 548us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:10:52.100591 (+ 1055us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:10:52.102664 (+ 2073us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:10:52.103384 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:10:52.103529 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:10:52.103979 (+ 450us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 02:10:52.104113 (+ 134us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 02:10:52.104197 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0928 02:10:52.104403 (+ 206us) server_negotiation.cc:299] Negotiation successful
0928 02:10:52.104491 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260928 02:10:52.104946 9446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:10:52.092326 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.235.65:46109 (local address 127.6.235.66:59153)
0928 02:10:52.098525 (+ 6199us) negotiation.cc:107] Waiting for socket to connect
0928 02:10:52.098545 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0928 02:10:52.098677 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:10:52.099019 (+ 342us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:10:52.099024 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:10:52.099036 (+ 12us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 02:10:52.099334 (+ 298us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:10:52.099344 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:10:52.100767 (+ 1423us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:10:52.100772 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:10:52.102468 (+ 1696us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:10:52.102482 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:10:52.103571 (+ 1089us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:10:52.103581 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:10:52.103728 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:10:52.103766 (+ 38us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 02:10:52.104248 (+ 482us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 02:10:52.104259 (+ 11us) client_negotiation.cc:769] Sending connection context
0928 02:10:52.104360 (+ 101us) client_negotiation.cc:240] Negotiation successful
0928 02:10:52.104507 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6037,"thread_start_us":5900,"threads_started":1}
I20260928 02:10:52.112514 9263 raft_consensus.cc:2454] T be5edf70978b428393a4d6c5b7e86425 P 8e4e2d7ffd0947f093e84367fdf816b3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1acdcdd55e604bc498460384fd246cb4 in term 1.
I20260928 02:10:52.113166 9264 raft_consensus.cc:1261] T be5edf70978b428393a4d6c5b7e86425 P 8e4e2d7ffd0947f093e84367fdf816b3 [term 1 FOLLOWER]: Refusing update from remote peer 1acdcdd55e604bc498460384fd246cb4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 02:10:52.116179 9431 consensus_queue.cc:1035] T be5edf70978b428393a4d6c5b7e86425 P 1acdcdd55e604bc498460384fd246cb4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8e4e2d7ffd0947f093e84367fdf816b3" member_type: VOTER last_known_addr { host: "127.6.235.67" port: 33761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 02:10:52.157353 8829 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:55846:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 02:10:52.157622 8829 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:55846:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 02:10:52.160953 8829 catalog_manager.cc:5782] T 00000000000000000000000000000000 P d52b1a21d6d74a8a8e0b264bdb19a5ab: Sending DeleteTablet for 3 replicas of tablet be5edf70978b428393a4d6c5b7e86425
I20260928 02:10:52.162019 9104 tablet_service.cc:1514] Processing DeleteTablet for tablet be5edf70978b428393a4d6c5b7e86425 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:10:52 UTC) from {username='kudu', principal='kudu/127.6.235.126@KRBTEST.COM'} at 127.0.0.1:59670
I20260928 02:10:52.162060 9244 tablet_service.cc:1514] Processing DeleteTablet for tablet be5edf70978b428393a4d6c5b7e86425 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:10:52 UTC) from {username='kudu', principal='kudu/127.6.235.126@KRBTEST.COM'} at 127.0.0.1:40886
I20260928 02:10:52.162331 9416 tablet_replica.cc:330] T be5edf70978b428393a4d6c5b7e86425 P 8e4e2d7ffd0947f093e84367fdf816b3: stopping tablet replica
I20260928 02:10:52.162326 9415 tablet_replica.cc:330] T be5edf70978b428393a4d6c5b7e86425 P 97e1a30f27ab4d1480891ee30a895fad: stopping tablet replica
I20260928 02:10:52.162518 9416 raft_consensus.cc:2229] T be5edf70978b428393a4d6c5b7e86425 P 8e4e2d7ffd0947f093e84367fdf816b3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 02:10:52.162518 9415 raft_consensus.cc:2229] T be5edf70978b428393a4d6c5b7e86425 P 97e1a30f27ab4d1480891ee30a895fad [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 02:10:52.162701 9416 raft_consensus.cc:2258] T be5edf70978b428393a4d6c5b7e86425 P 8e4e2d7ffd0947f093e84367fdf816b3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 02:10:52.162801 9415 raft_consensus.cc:2258] T be5edf70978b428393a4d6c5b7e86425 P 97e1a30f27ab4d1480891ee30a895fad [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 02:10:52.163553 8965 tablet_service.cc:1514] Processing DeleteTablet for tablet be5edf70978b428393a4d6c5b7e86425 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:10:52 UTC) from {username='kudu', principal='kudu/127.6.235.126@KRBTEST.COM'} at 127.0.0.1:52850
I20260928 02:10:52.163863 9414 tablet_replica.cc:330] T be5edf70978b428393a4d6c5b7e86425 P 1acdcdd55e604bc498460384fd246cb4: stopping tablet replica
I20260928 02:10:52.163635 8829 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:55846:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260928 02:10:52.164100 9414 raft_consensus.cc:2229] T be5edf70978b428393a4d6c5b7e86425 P 1acdcdd55e604bc498460384fd246cb4 [term 1 LEADER]: Raft consensus shutting down.
W20260928 02:10:52.164381 8829 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 02:10:52.164642 9414 raft_consensus.cc:2258] T be5edf70978b428393a4d6c5b7e86425 P 1acdcdd55e604bc498460384fd246cb4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 02:10:52.165936 9416 ts_tablet_manager.cc:1905] T be5edf70978b428393a4d6c5b7e86425 P 8e4e2d7ffd0947f093e84367fdf816b3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 02:10:52.166054 9415 ts_tablet_manager.cc:1905] T be5edf70978b428393a4d6c5b7e86425 P 97e1a30f27ab4d1480891ee30a895fad: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 02:10:52.169648 9414 ts_tablet_manager.cc:1905] T be5edf70978b428393a4d6c5b7e86425 P 1acdcdd55e604bc498460384fd246cb4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 02:10:52.175642 8965 tablet_service.cc:1467] Processing CreateTablet for tablet b1582d78b357403283139dbdfc93f1d6 (DEFAULT_TABLE table=test-table [id=084df1ea87d94ef298843fa6d02b6c5e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:10:52.175930 9104 tablet_service.cc:1467] Processing CreateTablet for tablet b1582d78b357403283139dbdfc93f1d6 (DEFAULT_TABLE table=test-table [id=084df1ea87d94ef298843fa6d02b6c5e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:10:52.176411 9244 tablet_service.cc:1467] Processing CreateTablet for tablet b1582d78b357403283139dbdfc93f1d6 (DEFAULT_TABLE table=test-table [id=084df1ea87d94ef298843fa6d02b6c5e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:10:52.220060 9104 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b1582d78b357403283139dbdfc93f1d6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:10:52.221356 8965 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b1582d78b357403283139dbdfc93f1d6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:10:52.224640 9244 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b1582d78b357403283139dbdfc93f1d6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:10:52.229898 9425 tablet_bootstrap.cc:492] T b1582d78b357403283139dbdfc93f1d6 P 97e1a30f27ab4d1480891ee30a895fad: Bootstrap starting.
I20260928 02:10:52.231616 9425 tablet_bootstrap.cc:654] T b1582d78b357403283139dbdfc93f1d6 P 97e1a30f27ab4d1480891ee30a895fad: Neither blocks nor log segments found. Creating new log.
I20260928 02:10:52.233675 9426 tablet_bootstrap.cc:492] T b1582d78b357403283139dbdfc93f1d6 P 1acdcdd55e604bc498460384fd246cb4: Bootstrap starting.
I20260928 02:10:52.234539 9425 tablet_bootstrap.cc:492] T b1582d78b357403283139dbdfc93f1d6 P 97e1a30f27ab4d1480891ee30a895fad: No bootstrap required, opened a new log
I20260928 02:10:52.234707 9425 ts_tablet_manager.cc:1397] T b1582d78b357403283139dbdfc93f1d6 P 97e1a30f27ab4d1480891ee30a895fad: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.001s
I20260928 02:10:52.235482 9426 tablet_bootstrap.cc:654] T b1582d78b357403283139dbdfc93f1d6 P 1acdcdd55e604bc498460384fd246cb4: Neither blocks nor log segments found. Creating new log.
I20260928 02:10:52.235512 9425 raft_consensus.cc:348] T b1582d78b357403283139dbdfc93f1d6 P 97e1a30f27ab4d1480891ee30a895fad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1acdcdd55e604bc498460384fd246cb4" member_type: VOTER last_known_addr { host: "127.6.235.65" port: 46109 } } peers { permanent_uuid: "97e1a30f27ab4d1480891ee30a895fad" member_type: VOTER last_known_addr { host: "127.6.235.66" port: 40129 } } peers { permanent_uuid: "8e4e2d7ffd0947f093e84367fdf816b3" member_type: VOTER last_known_addr { host: "127.6.235.67" port: 33761 } }
I20260928 02:10:52.235751 9425 raft_consensus.cc:374] T b1582d78b357403283139dbdfc93f1d6 P 97e1a30f27ab4d1480891ee30a895fad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:10:52.235845 9425 raft_consensus.cc:729] T b1582d78b357403283139dbdfc93f1d6 P 97e1a30f27ab4d1480891ee30a895fad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 97e1a30f27ab4d1480891ee30a895fad, State: Initialized, Role: FOLLOWER
I20260928 02:10:52.236104 9425 consensus_queue.cc:260] T b1582d78b357403283139dbdfc93f1d6 P 97e1a30f27ab4d1480891ee30a895fad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1acdcdd55e604bc498460384fd246cb4" member_type: VOTER last_known_addr { host: "127.6.235.65" port: 46109 } } peers { permanent_uuid: "97e1a30f27ab4d1480891ee30a895fad" member_type: VOTER last_known_addr { host: "127.6.235.66" port: 40129 } } peers { permanent_uuid: "8e4e2d7ffd0947f093e84367fdf816b3" member_type: VOTER last_known_addr { host: "127.6.235.67" port: 33761 } }
I20260928 02:10:52.237030 9425 ts_tablet_manager.cc:1428] T b1582d78b357403283139dbdfc93f1d6 P 97e1a30f27ab4d1480891ee30a895fad: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260928 02:10:52.238664 9426 tablet_bootstrap.cc:492] T b1582d78b357403283139dbdfc93f1d6 P 1acdcdd55e604bc498460384fd246cb4: No bootstrap required, opened a new log
I20260928 02:10:52.238858 9426 ts_tablet_manager.cc:1397] T b1582d78b357403283139dbdfc93f1d6 P 1acdcdd55e604bc498460384fd246cb4: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260928 02:10:52.239554 9426 raft_consensus.cc:348] T b1582d78b357403283139dbdfc93f1d6 P 1acdcdd55e604bc498460384fd246cb4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1acdcdd55e604bc498460384fd246cb4" member_type: VOTER last_known_addr { host: "127.6.235.65" port: 46109 } } peers { permanent_uuid: "97e1a30f27ab4d1480891ee30a895fad" member_type: VOTER last_known_addr { host: "127.6.235.66" port: 40129 } } peers { permanent_uuid: "8e4e2d7ffd0947f093e84367fdf816b3" member_type: VOTER last_known_addr { host: "127.6.235.67" port: 33761 } }
I20260928 02:10:52.239840 9426 raft_consensus.cc:374] T b1582d78b357403283139dbdfc93f1d6 P 1acdcdd55e604bc498460384fd246cb4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:10:52.239939 9426 raft_consensus.cc:729] T b1582d78b357403283139dbdfc93f1d6 P 1acdcdd55e604bc498460384fd246cb4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1acdcdd55e604bc498460384fd246cb4, State: Initialized, Role: FOLLOWER
I20260928 02:10:52.240200 9426 consensus_queue.cc:260] T b1582d78b357403283139dbdfc93f1d6 P 1acdcdd55e604bc498460384fd246cb4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1acdcdd55e604bc498460384fd246cb4" member_type: VOTER last_known_addr { host: "127.6.235.65" port: 46109 } } peers { permanent_uuid: "97e1a30f27ab4d1480891ee30a895fad" member_type: VOTER last_known_addr { host: "127.6.235.66" port: 40129 } } peers { permanent_uuid: "8e4e2d7ffd0947f093e84367fdf816b3" member_type: VOTER last_known_addr { host: "127.6.235.67" port: 33761 } }
I20260928 02:10:52.240828 9426 ts_tablet_manager.cc:1428] T b1582d78b357403283139dbdfc93f1d6 P 1acdcdd55e604bc498460384fd246cb4: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260928 02:10:52.318360 9416 ts_tablet_manager.cc:1918] T be5edf70978b428393a4d6c5b7e86425 P 8e4e2d7ffd0947f093e84367fdf816b3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 02:10:52.318588 9416 log.cc:1196] T be5edf70978b428393a4d6c5b7e86425 P 8e4e2d7ffd0947f093e84367fdf816b3: Deleting WAL directory at /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790561432251112-7085-0/minicluster-data/ts-2/wal/wals/be5edf70978b428393a4d6c5b7e86425
I20260928 02:10:52.319033 9416 ts_tablet_manager.cc:1939] T be5edf70978b428393a4d6c5b7e86425 P 8e4e2d7ffd0947f093e84367fdf816b3: Deleting consensus metadata
I20260928 02:10:52.323041 9414 ts_tablet_manager.cc:1918] T be5edf70978b428393a4d6c5b7e86425 P 1acdcdd55e604bc498460384fd246cb4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 02:10:52.323235 9414 log.cc:1196] T be5edf70978b428393a4d6c5b7e86425 P 1acdcdd55e604bc498460384fd246cb4: Deleting WAL directory at /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790561432251112-7085-0/minicluster-data/ts-0/wal/wals/be5edf70978b428393a4d6c5b7e86425
I20260928 02:10:52.323611 8815 catalog_manager.cc:4841] TS 8e4e2d7ffd0947f093e84367fdf816b3 (127.6.235.67:33761): tablet be5edf70978b428393a4d6c5b7e86425 (table test-table [id=c0c0268a5f5e406ea6e9f03a9545bc3f]) successfully deleted
I20260928 02:10:52.323699 9414 ts_tablet_manager.cc:1939] T be5edf70978b428393a4d6c5b7e86425 P 1acdcdd55e604bc498460384fd246cb4: Deleting consensus metadata
I20260928 02:10:52.324025 9427 tablet_bootstrap.cc:492] T b1582d78b357403283139dbdfc93f1d6 P 8e4e2d7ffd0947f093e84367fdf816b3: Bootstrap starting.
I20260928 02:10:52.324489 9415 ts_tablet_manager.cc:1918] T be5edf70978b428393a4d6c5b7e86425 P 97e1a30f27ab4d1480891ee30a895fad: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 02:10:52.324738 9415 log.cc:1196] T be5edf70978b428393a4d6c5b7e86425 P 97e1a30f27ab4d1480891ee30a895fad: Deleting WAL directory at /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790561432251112-7085-0/minicluster-data/ts-1/wal/wals/be5edf70978b428393a4d6c5b7e86425
I20260928 02:10:52.325271 9415 ts_tablet_manager.cc:1939] T be5edf70978b428393a4d6c5b7e86425 P 97e1a30f27ab4d1480891ee30a895fad: Deleting consensus metadata
I20260928 02:10:52.325726 9427 tablet_bootstrap.cc:654] T b1582d78b357403283139dbdfc93f1d6 P 8e4e2d7ffd0947f093e84367fdf816b3: Neither blocks nor log segments found. Creating new log.
I20260928 02:10:52.329464 9427 tablet_bootstrap.cc:492] T b1582d78b357403283139dbdfc93f1d6 P 8e4e2d7ffd0947f093e84367fdf816b3: No bootstrap required, opened a new log
I20260928 02:10:52.329658 9427 ts_tablet_manager.cc:1397] T b1582d78b357403283139dbdfc93f1d6 P 8e4e2d7ffd0947f093e84367fdf816b3: Time spent bootstrapping tablet: real 0.006s user 0.003s sys 0.001s
I20260928 02:10:52.330456 9427 raft_consensus.cc:348] T b1582d78b357403283139dbdfc93f1d6 P 8e4e2d7ffd0947f093e84367fdf816b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1acdcdd55e604bc498460384fd246cb4" member_type: VOTER last_known_addr { host: "127.6.235.65" port: 46109 } } peers { permanent_uuid: "97e1a30f27ab4d1480891ee30a895fad" member_type: VOTER last_known_addr { host: "127.6.235.66" port: 40129 } } peers { permanent_uuid: "8e4e2d7ffd0947f093e84367fdf816b3" member_type: VOTER last_known_addr { host: "127.6.235.67" port: 33761 } }
I20260928 02:10:52.330689 9427 raft_consensus.cc:374] T b1582d78b357403283139dbdfc93f1d6 P 8e4e2d7ffd0947f093e84367fdf816b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:10:52.330770 9427 raft_consensus.cc:729] T b1582d78b357403283139dbdfc93f1d6 P 8e4e2d7ffd0947f093e84367fdf816b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8e4e2d7ffd0947f093e84367fdf816b3, State: Initialized, Role: FOLLOWER
I20260928 02:10:52.331030 9427 consensus_queue.cc:260] T b1582d78b357403283139dbdfc93f1d6 P 8e4e2d7ffd0947f093e84367fdf816b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1acdcdd55e604bc498460384fd246cb4" member_type: VOTER last_known_addr { host: "127.6.235.65" port: 46109 } } peers { permanent_uuid: "97e1a30f27ab4d1480891ee30a895fad" member_type: VOTER last_known_addr { host: "127.6.235.66" port: 40129 } } peers { permanent_uuid: "8e4e2d7ffd0947f093e84367fdf816b3" member_type: VOTER last_known_addr { host: "127.6.235.67" port: 33761 } }
I20260928 02:10:52.331534 9427 ts_tablet_manager.cc:1428] T b1582d78b357403283139dbdfc93f1d6 P 8e4e2d7ffd0947f093e84367fdf816b3: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20260928 02:10:52.418222 8814 catalog_manager.cc:4841] TS 97e1a30f27ab4d1480891ee30a895fad (127.6.235.66:40129): tablet be5edf70978b428393a4d6c5b7e86425 (table test-table [id=c0c0268a5f5e406ea6e9f03a9545bc3f]) successfully deleted
I20260928 02:10:52.418223 8815 catalog_manager.cc:4841] TS 1acdcdd55e604bc498460384fd246cb4 (127.6.235.65:46109): tablet be5edf70978b428393a4d6c5b7e86425 (table test-table [id=c0c0268a5f5e406ea6e9f03a9545bc3f]) successfully deleted
I20260928 02:10:52.488840 9432 raft_consensus.cc:482] T b1582d78b357403283139dbdfc93f1d6 P 8e4e2d7ffd0947f093e84367fdf816b3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 02:10:52.489255 9432 raft_consensus.cc:504] T b1582d78b357403283139dbdfc93f1d6 P 8e4e2d7ffd0947f093e84367fdf816b3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1acdcdd55e604bc498460384fd246cb4" member_type: VOTER last_known_addr { host: "127.6.235.65" port: 46109 } } peers { permanent_uuid: "97e1a30f27ab4d1480891ee30a895fad" member_type: VOTER last_known_addr { host: "127.6.235.66" port: 40129 } } peers { permanent_uuid: "8e4e2d7ffd0947f093e84367fdf816b3" member_type: VOTER last_known_addr { host: "127.6.235.67" port: 33761 } }
I20260928 02:10:52.489975 9432 leader_election.cc:288] T b1582d78b357403283139dbdfc93f1d6 P 8e4e2d7ffd0947f093e84367fdf816b3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1acdcdd55e604bc498460384fd246cb4 (127.6.235.65:46109), 97e1a30f27ab4d1480891ee30a895fad (127.6.235.66:40129)
I20260928 02:10:52.490542 9128 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b1582d78b357403283139dbdfc93f1d6" candidate_uuid: "8e4e2d7ffd0947f093e84367fdf816b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "97e1a30f27ab4d1480891ee30a895fad" is_pre_election: true
I20260928 02:10:52.490540 8985 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b1582d78b357403283139dbdfc93f1d6" candidate_uuid: "8e4e2d7ffd0947f093e84367fdf816b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1acdcdd55e604bc498460384fd246cb4" is_pre_election: true
I20260928 02:10:52.490803 9128 raft_consensus.cc:2454] T b1582d78b357403283139dbdfc93f1d6 P 97e1a30f27ab4d1480891ee30a895fad [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8e4e2d7ffd0947f093e84367fdf816b3 in term 0.
I20260928 02:10:52.490825 8985 raft_consensus.cc:2454] T b1582d78b357403283139dbdfc93f1d6 P 1acdcdd55e604bc498460384fd246cb4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8e4e2d7ffd0947f093e84367fdf816b3 in term 0.
I20260928 02:10:52.491312 9196 leader_election.cc:302] T b1582d78b357403283139dbdfc93f1d6 P 8e4e2d7ffd0947f093e84367fdf816b3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8e4e2d7ffd0947f093e84367fdf816b3, 97e1a30f27ab4d1480891ee30a895fad; no voters:
I20260928 02:10:52.491667 9432 raft_consensus.cc:2781] T b1582d78b357403283139dbdfc93f1d6 P 8e4e2d7ffd0947f093e84367fdf816b3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 02:10:52.491793 9432 raft_consensus.cc:482] T b1582d78b357403283139dbdfc93f1d6 P 8e4e2d7ffd0947f093e84367fdf816b3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 02:10:52.491884 9432 raft_consensus.cc:3037] T b1582d78b357403283139dbdfc93f1d6 P 8e4e2d7ffd0947f093e84367fdf816b3 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:10:52.523815 9432 raft_consensus.cc:504] T b1582d78b357403283139dbdfc93f1d6 P 8e4e2d7ffd0947f093e84367fdf816b3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1acdcdd55e604bc498460384fd246cb4" member_type: VOTER last_known_addr { host: "127.6.235.65" port: 46109 } } peers { permanent_uuid: "97e1a30f27ab4d1480891ee30a895fad" member_type: VOTER last_known_addr { host: "127.6.235.66" port: 40129 } } peers { permanent_uuid: "8e4e2d7ffd0947f093e84367fdf816b3" member_type: VOTER last_known_addr { host: "127.6.235.67" port: 33761 } }
I20260928 02:10:52.524652 9432 leader_election.cc:288] T b1582d78b357403283139dbdfc93f1d6 P 8e4e2d7ffd0947f093e84367fdf816b3 [CANDIDATE]: Term 1 election: Requested vote from peers 1acdcdd55e604bc498460384fd246cb4 (127.6.235.65:46109), 97e1a30f27ab4d1480891ee30a895fad (127.6.235.66:40129)
I20260928 02:10:52.525449 8985 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b1582d78b357403283139dbdfc93f1d6" candidate_uuid: "8e4e2d7ffd0947f093e84367fdf816b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1acdcdd55e604bc498460384fd246cb4"
I20260928 02:10:52.525552 9128 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b1582d78b357403283139dbdfc93f1d6" candidate_uuid: "8e4e2d7ffd0947f093e84367fdf816b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "97e1a30f27ab4d1480891ee30a895fad"
I20260928 02:10:52.525712 8985 raft_consensus.cc:3037] T b1582d78b357403283139dbdfc93f1d6 P 1acdcdd55e604bc498460384fd246cb4 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:10:52.525774 9128 raft_consensus.cc:3037] T b1582d78b357403283139dbdfc93f1d6 P 97e1a30f27ab4d1480891ee30a895fad [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:10:52.528935 9128 raft_consensus.cc:2454] T b1582d78b357403283139dbdfc93f1d6 P 97e1a30f27ab4d1480891ee30a895fad [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8e4e2d7ffd0947f093e84367fdf816b3 in term 1.
I20260928 02:10:52.529429 8985 raft_consensus.cc:2454] T b1582d78b357403283139dbdfc93f1d6 P 1acdcdd55e604bc498460384fd246cb4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8e4e2d7ffd0947f093e84367fdf816b3 in term 1.
I20260928 02:10:52.529587 9196 leader_election.cc:302] T b1582d78b357403283139dbdfc93f1d6 P 8e4e2d7ffd0947f093e84367fdf816b3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8e4e2d7ffd0947f093e84367fdf816b3, 97e1a30f27ab4d1480891ee30a895fad; no voters:
I20260928 02:10:52.530090 9432 raft_consensus.cc:2781] T b1582d78b357403283139dbdfc93f1d6 P 8e4e2d7ffd0947f093e84367fdf816b3 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:10:52.530517 9432 raft_consensus.cc:686] T b1582d78b357403283139dbdfc93f1d6 P 8e4e2d7ffd0947f093e84367fdf816b3 [term 1 LEADER]: Becoming Leader. State: Replica: 8e4e2d7ffd0947f093e84367fdf816b3, State: Running, Role: LEADER
I20260928 02:10:52.530823 9432 consensus_queue.cc:237] T b1582d78b357403283139dbdfc93f1d6 P 8e4e2d7ffd0947f093e84367fdf816b3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1acdcdd55e604bc498460384fd246cb4" member_type: VOTER last_known_addr { host: "127.6.235.65" port: 46109 } } peers { permanent_uuid: "97e1a30f27ab4d1480891ee30a895fad" member_type: VOTER last_known_addr { host: "127.6.235.66" port: 40129 } } peers { permanent_uuid: "8e4e2d7ffd0947f093e84367fdf816b3" member_type: VOTER last_known_addr { host: "127.6.235.67" port: 33761 } }
I20260928 02:10:52.533856 8829 catalog_manager.cc:5495] T b1582d78b357403283139dbdfc93f1d6 P 8e4e2d7ffd0947f093e84367fdf816b3 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8e4e2d7ffd0947f093e84367fdf816b3 (127.6.235.67). New cstate: current_term: 1 leader_uuid: "8e4e2d7ffd0947f093e84367fdf816b3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1acdcdd55e604bc498460384fd246cb4" member_type: VOTER last_known_addr { host: "127.6.235.65" port: 46109 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "97e1a30f27ab4d1480891ee30a895fad" member_type: VOTER last_known_addr { host: "127.6.235.66" port: 40129 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8e4e2d7ffd0947f093e84367fdf816b3" member_type: VOTER last_known_addr { host: "127.6.235.67" port: 33761 } health_report { overall_health: HEALTHY } } }
I20260928 02:10:52.578176 9458 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:10:52.563256 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41626 (local address 127.6.235.67:33761)
0928 02:10:52.572315 (+ 9059us) server_negotiation.cc:206] Beginning negotiation
0928 02:10:52.572320 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 02:10:52.572344 (+ 24us) server_negotiation.cc:407] Connection header received
0928 02:10:52.572411 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:10:52.572415 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:10:52.572498 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:10:52.572607 (+ 109us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 02:10:52.573294 (+ 687us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:10:52.574079 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:10:52.576229 (+ 2150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:10:52.576661 (+ 432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:10:52.576763 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:10:52.577278 (+ 515us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 02:10:52.577402 (+ 124us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 02:10:52.577495 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0928 02:10:52.577715 (+ 220us) server_negotiation.cc:299] Negotiation successful
0928 02:10:52.577843 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8926,"thread_start_us":8798,"threads_started":1}
I20260928 02:10:52.588418 8985 raft_consensus.cc:1261] T b1582d78b357403283139dbdfc93f1d6 P 1acdcdd55e604bc498460384fd246cb4 [term 1 FOLLOWER]: Refusing update from remote peer 8e4e2d7ffd0947f093e84367fdf816b3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 02:10:52.588524 9128 raft_consensus.cc:1261] T b1582d78b357403283139dbdfc93f1d6 P 97e1a30f27ab4d1480891ee30a895fad [term 1 FOLLOWER]: Refusing update from remote peer 8e4e2d7ffd0947f093e84367fdf816b3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 02:10:52.588989 9454 consensus_queue.cc:1035] T b1582d78b357403283139dbdfc93f1d6 P 8e4e2d7ffd0947f093e84367fdf816b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1acdcdd55e604bc498460384fd246cb4" member_type: VOTER last_known_addr { host: "127.6.235.65" port: 46109 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 02:10:52.589385 9432 consensus_queue.cc:1035] T b1582d78b357403283139dbdfc93f1d6 P 8e4e2d7ffd0947f093e84367fdf816b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "97e1a30f27ab4d1480891ee30a895fad" member_type: VOTER last_known_addr { host: "127.6.235.66" port: 40129 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260928 02:10:52.608274 9398 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260928 02:10:54.198319 8829 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:55846:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 02:10:54.198625 8829 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:55846:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 02:10:54.203060 8829 catalog_manager.cc:5782] T 00000000000000000000000000000000 P d52b1a21d6d74a8a8e0b264bdb19a5ab: Sending DeleteTablet for 3 replicas of tablet b1582d78b357403283139dbdfc93f1d6
I20260928 02:10:54.204068 9244 tablet_service.cc:1514] Processing DeleteTablet for tablet b1582d78b357403283139dbdfc93f1d6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:10:54 UTC) from {username='kudu', principal='kudu/127.6.235.126@KRBTEST.COM'} at 127.0.0.1:40886
I20260928 02:10:54.204124 8965 tablet_service.cc:1514] Processing DeleteTablet for tablet b1582d78b357403283139dbdfc93f1d6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:10:54 UTC) from {username='kudu', principal='kudu/127.6.235.126@KRBTEST.COM'} at 127.0.0.1:52850
I20260928 02:10:54.204104 9104 tablet_service.cc:1514] Processing DeleteTablet for tablet b1582d78b357403283139dbdfc93f1d6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:10:54 UTC) from {username='kudu', principal='kudu/127.6.235.126@KRBTEST.COM'} at 127.0.0.1:59670
I20260928 02:10:54.204716 7085 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskTlaUuc/build/asan/bin/kudu pid 8900
I20260928 02:10:54.218314 9486 tablet_replica.cc:330] T b1582d78b357403283139dbdfc93f1d6 P 8e4e2d7ffd0947f093e84367fdf816b3: stopping tablet replica
I20260928 02:10:54.218544 9486 raft_consensus.cc:2229] T b1582d78b357403283139dbdfc93f1d6 P 8e4e2d7ffd0947f093e84367fdf816b3 [term 1 LEADER]: Raft consensus shutting down.
I20260928 02:10:54.219004 9486 raft_consensus.cc:2258] T b1582d78b357403283139dbdfc93f1d6 P 8e4e2d7ffd0947f093e84367fdf816b3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 02:10:54.220113 9486 ts_tablet_manager.cc:1905] T b1582d78b357403283139dbdfc93f1d6 P 8e4e2d7ffd0947f093e84367fdf816b3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 02:10:54.223922 9488 tablet_replica.cc:330] T b1582d78b357403283139dbdfc93f1d6 P 1acdcdd55e604bc498460384fd246cb4: stopping tablet replica
I20260928 02:10:54.224134 9488 raft_consensus.cc:2229] T b1582d78b357403283139dbdfc93f1d6 P 1acdcdd55e604bc498460384fd246cb4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 02:10:54.224265 9487 tablet_replica.cc:330] T b1582d78b357403283139dbdfc93f1d6 P 97e1a30f27ab4d1480891ee30a895fad: stopping tablet replica
I20260928 02:10:54.224440 9488 raft_consensus.cc:2258] T b1582d78b357403283139dbdfc93f1d6 P 1acdcdd55e604bc498460384fd246cb4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 02:10:54.224493 9487 raft_consensus.cc:2229] T b1582d78b357403283139dbdfc93f1d6 P 97e1a30f27ab4d1480891ee30a895fad [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 02:10:54.224778 9487 raft_consensus.cc:2258] T b1582d78b357403283139dbdfc93f1d6 P 97e1a30f27ab4d1480891ee30a895fad [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 02:10:54.225855 9488 ts_tablet_manager.cc:1905] T b1582d78b357403283139dbdfc93f1d6 P 1acdcdd55e604bc498460384fd246cb4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 02:10:54.226027 9487 ts_tablet_manager.cc:1905] T b1582d78b357403283139dbdfc93f1d6 P 97e1a30f27ab4d1480891ee30a895fad: Deleting tablet data with delete state TABLET_DATA_DELETED
Sep 28 02:10:54 dist-test-slave-86gv krb5kdc[8784](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561451, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.6.235.65@KRBTEST.COM
I20260928 02:10:54.235194 9489 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:10:54.205928 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41538 (local address 127.6.235.65:46109)
0928 02:10:54.216780 (+ 10852us) server_negotiation.cc:206] Beginning negotiation
0928 02:10:54.216787 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 02:10:54.216816 (+ 29us) server_negotiation.cc:407] Connection header received
0928 02:10:54.216915 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:10:54.216924 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:10:54.217057 (+ 133us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:10:54.217187 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:10:54.219564 (+ 2377us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:10:54.220742 (+ 1178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:10:54.222737 (+ 1995us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:10:54.223436 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:10:54.223617 (+ 181us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:10:54.229463 (+ 5846us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:10:54.229479 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:10:54.229495 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:10:54.229535 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:10:54.232233 (+ 2698us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:10:54.232939 (+ 706us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:10:54.232951 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:10:54.232956 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:10:54.233150 (+ 194us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:10:54.233549 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:10:54.233554 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:10:54.233557 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:10:54.234006 (+ 449us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:10:54.234302 (+ 296us) server_negotiation.cc:1035] Waiting for connection context
0928 02:10:54.234600 (+ 298us) server_negotiation.cc:299] Negotiation successful
0928 02:10:54.234817 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10687,"thread_start_us":11383,"threads_started":1}
W20260928 02:10:54.235881 9031 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:41538
W20260928 02:10:54.236521 7085 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskTlaUuc/build/asan/bin/kudu pid 8900: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260928 02:10:54.236734 7085 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTlaUuc/build/asan/bin/kudu with pid 8900
I20260928 02:10:54.327481 9486 ts_tablet_manager.cc:1918] T b1582d78b357403283139dbdfc93f1d6 P 8e4e2d7ffd0947f093e84367fdf816b3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260928 02:10:54.327550 9487 ts_tablet_manager.cc:1918] T b1582d78b357403283139dbdfc93f1d6 P 97e1a30f27ab4d1480891ee30a895fad: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260928 02:10:54.327647 9486 log.cc:1196] T b1582d78b357403283139dbdfc93f1d6 P 8e4e2d7ffd0947f093e84367fdf816b3: Deleting WAL directory at /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790561432251112-7085-0/minicluster-data/ts-2/wal/wals/b1582d78b357403283139dbdfc93f1d6
I20260928 02:10:54.327692 9487 log.cc:1196] T b1582d78b357403283139dbdfc93f1d6 P 97e1a30f27ab4d1480891ee30a895fad: Deleting WAL directory at /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790561432251112-7085-0/minicluster-data/ts-1/wal/wals/b1582d78b357403283139dbdfc93f1d6
I20260928 02:10:54.328090 9486 ts_tablet_manager.cc:1939] T b1582d78b357403283139dbdfc93f1d6 P 8e4e2d7ffd0947f093e84367fdf816b3: Deleting consensus metadata
I20260928 02:10:54.328132 9487 ts_tablet_manager.cc:1939] T b1582d78b357403283139dbdfc93f1d6 P 97e1a30f27ab4d1480891ee30a895fad: Deleting consensus metadata
I20260928 02:10:54.348181 7085 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskTlaUuc/build/asan/bin/kudu pid 9040
W20260928 02:10:54.348196 8815 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.6.235.65:46109 (error 108)
I20260928 02:10:54.350836 9493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:10:54.349890 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.235.65:46109 (local address 127.0.0.1:41550)
0928 02:10:54.350445 (+ 555us) negotiation.cc:107] Waiting for socket to connect
0928 02:10:54.350581 (+ 136us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.235.65:46109: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":416,"thread_start_us":272,"threads_started":1}
W20260928 02:10:54.351051 8815 catalog_manager.cc:4584] TS 1acdcdd55e604bc498460384fd246cb4 (127.6.235.65:46109): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet b1582d78b357403283139dbdfc93f1d6: Network error: Client connection negotiation failed: client connection to 127.6.235.65:46109: connect: Connection refused (error 111)
Sep 28 02:10:54 dist-test-slave-86gv krb5kdc[8784](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561451, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.6.235.66@KRBTEST.COM
I20260928 02:10:54.370250 9492 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:10:54.349144 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37610 (local address 127.6.235.66:40129)
0928 02:10:54.349889 (+ 745us) server_negotiation.cc:206] Beginning negotiation
0928 02:10:54.349894 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 02:10:54.349933 (+ 39us) server_negotiation.cc:407] Connection header received
0928 02:10:54.349988 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:10:54.349991 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:10:54.350081 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:10:54.350208 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:10:54.351778 (+ 1570us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:10:54.352534 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:10:54.354190 (+ 1656us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:10:54.354621 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:10:54.354733 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:10:54.364148 (+ 9415us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:10:54.364166 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:10:54.364183 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:10:54.364229 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:10:54.367440 (+ 3211us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:10:54.368169 (+ 729us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:10:54.368178 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:10:54.368180 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:10:54.368307 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:10:54.368665 (+ 358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:10:54.368669 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:10:54.368671 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:10:54.369109 (+ 438us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:10:54.369373 (+ 264us) server_negotiation.cc:1035] Waiting for connection context
0928 02:10:54.369655 (+ 282us) server_negotiation.cc:299] Negotiation successful
0928 02:10:54.369827 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":603,"thread_start_us":437,"threads_started":1}
W20260928 02:10:54.370924 9170 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:37610
W20260928 02:10:54.371587 7085 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskTlaUuc/build/asan/bin/kudu pid 9040: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260928 02:10:54.371742 7085 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTlaUuc/build/asan/bin/kudu with pid 9040
I20260928 02:10:54.404594 9493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:10:54.404140 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.235.65:46109 (local address 127.0.0.1:41562)
0928 02:10:54.404346 (+ 206us) negotiation.cc:107] Waiting for socket to connect
0928 02:10:54.404444 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.235.65:46109: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":71}
I20260928 02:10:54.406214 9493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:10:54.405305 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.235.65:46109 (local address 127.0.0.1:41570)
0928 02:10:54.405491 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0928 02:10:54.406073 (+ 582us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.235.65:46109: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
I20260928 02:10:54.415019 8815 catalog_manager.cc:4841] TS 8e4e2d7ffd0947f093e84367fdf816b3 (127.6.235.67:33761): tablet b1582d78b357403283139dbdfc93f1d6 (table test-table [id=084df1ea87d94ef298843fa6d02b6c5e]) successfully deleted
I20260928 02:10:54.441390 7085 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskTlaUuc/build/asan/bin/kudu pid 9179
I20260928 02:10:54.442160 9493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:10:54.441638 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.235.66:40129 (local address 127.0.0.1:37618)
0928 02:10:54.441833 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0928 02:10:54.441927 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.235.66:40129: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
Sep 28 02:10:54 dist-test-slave-86gv krb5kdc[8784](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561451, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.6.235.67@KRBTEST.COM
I20260928 02:10:54.457695 9494 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:10:54.442170 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41632 (local address 127.6.235.67:33761)
0928 02:10:54.442809 (+ 639us) server_negotiation.cc:206] Beginning negotiation
0928 02:10:54.442813 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:10:54.442830 (+ 17us) server_negotiation.cc:407] Connection header received
0928 02:10:54.442888 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:10:54.442892 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:10:54.442957 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:10:54.443069 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:10:54.444551 (+ 1482us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:10:54.445341 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:10:54.446884 (+ 1543us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:10:54.447303 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:10:54.447399 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:10:54.452514 (+ 5115us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:10:54.452531 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:10:54.452548 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:10:54.452585 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:10:54.455380 (+ 2795us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:10:54.456022 (+ 642us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:10:54.456028 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:10:54.456030 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:10:54.456147 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:10:54.456450 (+ 303us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:10:54.456453 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:10:54.456455 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:10:54.456777 (+ 322us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:10:54.457093 (+ 316us) server_negotiation.cc:1035] Waiting for connection context
0928 02:10:54.457280 (+ 187us) server_negotiation.cc:299] Negotiation successful
0928 02:10:54.457444 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":528,"thread_start_us":396,"threads_started":1}
W20260928 02:10:54.458372 9309 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:41632
W20260928 02:10:54.459059 7085 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskTlaUuc/build/asan/bin/kudu pid 9179: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260928 02:10:54.459185 7085 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTlaUuc/build/asan/bin/kudu with pid 9179
I20260928 02:10:54.467545 9493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:10:54.466894 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.235.65:46109 (local address 127.0.0.1:41578)
0928 02:10:54.467151 (+ 257us) negotiation.cc:107] Waiting for socket to connect
0928 02:10:54.467306 (+ 155us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.235.65:46109: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":85}
I20260928 02:10:54.468715 9493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:10:54.468271 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.235.65:46109 (local address 127.0.0.1:41592)
0928 02:10:54.468461 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0928 02:10:54.468555 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.235.65:46109: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
I20260928 02:10:54.475418 9493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:10:54.474842 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.235.66:40129 (local address 127.0.0.1:37624)
0928 02:10:54.475115 (+ 273us) negotiation.cc:107] Waiting for socket to connect
0928 02:10:54.475223 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.235.66:40129: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":86}
I20260928 02:10:54.476580 9493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:10:54.476122 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.235.66:40129 (local address 127.0.0.1:37636)
0928 02:10:54.476332 (+ 210us) negotiation.cc:107] Waiting for socket to connect
0928 02:10:54.476431 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.235.66:40129: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
I20260928 02:10:54.482933 7085 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskTlaUuc/build/asan/bin/kudu pid 8797
I20260928 02:10:54.484045 8869 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 02:10:54.600239 9493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:10:54.599700 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.235.66:40129 (local address 127.0.0.1:37650)
0928 02:10:54.599889 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0928 02:10:54.599974 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.235.66:40129: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
I20260928 02:10:54.600883 9493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:10:54.600273 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.235.65:46109 (local address 127.0.0.1:41604)
0928 02:10:54.600506 (+ 233us) negotiation.cc:107] Waiting for socket to connect
0928 02:10:54.600608 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.235.65:46109: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":141,"thread_start_us":338,"threads_started":1}
I20260928 02:10:54.601147 7085 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTlaUuc/build/asan/bin/kudu with pid 8797
2026-09-28T02:10:54Z chronyd exiting
[ OK ] SecurityITest.TxnSmokeWithDifferentUserTypes (9328 ms)
[ RUN ] SecurityITest.TestNoKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790561432251112-7085-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 02:10:55 dist-test-slave-86gv krb5kdc[9499](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 02:10:55 dist-test-slave-86gv krb5kdc[9499](info): set up 2 sockets
Sep 28 02:10:55 dist-test-slave-86gv krb5kdc[9499](info): commencing operation
krb5kdc: starting...
W20260928 02:10:56.021656 7085 mini_kdc.cc:121] Time spent starting KDC: real 1.308s user 0.004s sys 0.015s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790561432251112-7085-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790561432251112-7085-0/krb5kdc/test-user.keytab.
Sep 28 02:10:56 dist-test-slave-86gv krb5kdc[9499](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561456, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T02:10:56Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T02:10:56Z Disabled control of system clock
WARNING: no policy specified for kudu/127.6.235.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.235.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.235.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790561432251112-7085-0/krb5kdc/kudu_127.6.235.126.keytab.
Entry for principal kudu/127.6.235.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790561432251112-7085-0/krb5kdc/kudu_127.6.235.126.keytab.
I20260928 02:10:56.233024 7085 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTlaUuc/build/asan/bin/kudu
/tmp/dist-test-taskTlaUuc/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790561432251112-7085-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790561432251112-7085-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790561432251112-7085-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790561432251112-7085-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.235.126:43159
--webserver_interface=127.6.235.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790561432251112-7085-0/krb5kdc/kudu_127.6.235.126.keytab
--principal=kudu/127.6.235.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.235.84:39389
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.235.126:43159
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790561432251112-7085-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790561432251112-7085-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790561432251112-7085-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:10:56.585901 9512 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:10:56.586210 9512 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:10:56.586386 9512 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:10:56.593844 9512 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 02:10:56.593966 9512 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:10:56.594065 9512 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:10:56.594151 9512 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 02:10:56.594237 9512 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 02:10:56.604269 9512 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.235.84:39389
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790561432251112-7085-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790561432251112-7085-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.235.126:43159
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790561432251112-7085-0/krb5kdc/kudu_127.6.235.126.keytab
--principal=kudu/127.6.235.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.235.126:43159
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790561432251112-7085-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.235.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790561432251112-7085-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 00:51:59 UTC on ccycloud.quasar-axgkwn.root.comops.site
build id 4676464
ASAN enabled
I20260928 02:10:56.605710 9512 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:10:56.607090 9512 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:10:56.619676 9517 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:10:56.619741 9520 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:10:56.619947 9518 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:10:56.620489 9512 server_base.cc:1034] running on GCE node
I20260928 02:10:56.621381 9512 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:10:56.623003 9512 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:10:56.624313 9512 hybrid_clock.cc:648] HybridClock initialized: now 1790561456624201 us; error 105 us; skew 500 ppm
I20260928 02:10:56.624776 9512 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 02:10:56 dist-test-slave-86gv krb5kdc[9499](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561456, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.235.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:10:56.628507 9512 init.cc:377] Logged in from keytab as kudu/127.6.235.126@KRBTEST.COM (short username kudu)
I20260928 02:10:56.630286 9512 webserver.cc:460] Webserver started at http://127.6.235.126:42595/ using document root <none> and password file <none>
I20260928 02:10:56.631074 9512 fs_manager.cc:359] Metadata directory not provided
I20260928 02:10:56.631186 9512 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:10:56.631481 9512 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:10:56.634931 9512 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790561432251112-7085-0/minicluster-data/master-0/data/instance:
uuid: "f8aeaa8649df40b3ad3523c88291d27f"
format_stamp: "Formatted at 2026-09-28 02:10:56 on dist-test-slave-86gv"
server_key: "baf98922c69fa7345d2dcf4c88aafb08"
server_key_iv: "b9fc4f48f3cc24498a212d32bd149ee1"
server_key_version: "encryptionkey@0"
I20260928 02:10:56.635758 9512 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790561432251112-7085-0/minicluster-data/master-0/wal/instance:
uuid: "f8aeaa8649df40b3ad3523c88291d27f"
format_stamp: "Formatted at 2026-09-28 02:10:56 on dist-test-slave-86gv"
server_key: "baf98922c69fa7345d2dcf4c88aafb08"
server_key_iv: "b9fc4f48f3cc24498a212d32bd149ee1"
server_key_version: "encryptionkey@0"
I20260928 02:10:56.641477 9512 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260928 02:10:56.647568 9527 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:10:56.649549 9512 fs_manager.cc:725] Time spent opening block manager: real 0.006s user 0.007s sys 0.000s
I20260928 02:10:56.649796 9512 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790561432251112-7085-0/minicluster-data/master-0/data,/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790561432251112-7085-0/minicluster-data/master-0/wal
uuid: "f8aeaa8649df40b3ad3523c88291d27f"
format_stamp: "Formatted at 2026-09-28 02:10:56 on dist-test-slave-86gv"
server_key: "baf98922c69fa7345d2dcf4c88aafb08"
server_key_iv: "b9fc4f48f3cc24498a212d32bd149ee1"
server_key_version: "encryptionkey@0"
I20260928 02:10:56.650060 9512 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790561432251112-7085-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790561432251112-7085-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790561432251112-7085-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:10:56.663079 9512 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:10:56.667698 9512 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:10:56.667959 9512 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:10:56.687559 9512 rpc_server.cc:307] RPC server started. Bound to: 127.6.235.126:43159
I20260928 02:10:56.687592 9578 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.235.126:43159 every 8 connection(s)
I20260928 02:10:56.689357 9512 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790561432251112-7085-0/minicluster-data/master-0/data/info.pb
I20260928 02:10:56.693334 7085 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTlaUuc/build/asan/bin/kudu as pid 9512
I20260928 02:10:56.693534 7085 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790561432251112-7085-0/minicluster-data/master-0/wal/instance
I20260928 02:10:56.693990 9579 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:10:56.694222 7085 external_mini_cluster.cc:1428] Setting key 90d3a308ecb58d1e7707e566a280d122
I20260928 02:10:56.708665 9579 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f8aeaa8649df40b3ad3523c88291d27f: Bootstrap starting.
Sep 28 02:10:56 dist-test-slave-86gv krb5kdc[9499](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561456, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.235.126@KRBTEST.COM
I20260928 02:10:56.712826 9579 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f8aeaa8649df40b3ad3523c88291d27f: Neither blocks nor log segments found. Creating new log.
I20260928 02:10:56.714290 9579 log.cc:824] T 00000000000000000000000000000000 P f8aeaa8649df40b3ad3523c88291d27f: Log is configured to *not* fsync() on all Append() calls
I20260928 02:10:56.717495 9579 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f8aeaa8649df40b3ad3523c88291d27f: No bootstrap required, opened a new log
I20260928 02:10:56.718817 9582 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:10:56.696287 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57638 (local address 127.6.235.126:43159)
0928 02:10:56.697134 (+ 847us) server_negotiation.cc:206] Beginning negotiation
0928 02:10:56.697154 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0928 02:10:56.697204 (+ 50us) server_negotiation.cc:407] Connection header received
0928 02:10:56.699004 (+ 1800us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:10:56.699038 (+ 34us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:10:56.699592 (+ 554us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:10:56.700126 (+ 534us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:10:56.701529 (+ 1403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:10:56.703544 (+ 2015us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:10:56.705132 (+ 1588us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:10:56.706280 (+ 1148us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:10:56.706474 (+ 194us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:10:56.710684 (+ 4210us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:10:56.710695 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:10:56.710717 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:10:56.710774 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:10:56.714712 (+ 3938us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:10:56.715476 (+ 764us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:10:56.715483 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:10:56.715489 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:10:56.715671 (+ 182us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:10:56.716066 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:10:56.716071 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:10:56.716076 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:10:56.716621 (+ 545us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:10:56.716941 (+ 320us) server_negotiation.cc:1035] Waiting for connection context
0928 02:10:56.717380 (+ 439us) server_negotiation.cc:299] Negotiation successful
0928 02:10:56.717840 (+ 460us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":560,"thread_start_us":312,"threads_started":1}
I20260928 02:10:56.725771 9579 raft_consensus.cc:348] T 00000000000000000000000000000000 P f8aeaa8649df40b3ad3523c88291d27f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f8aeaa8649df40b3ad3523c88291d27f" member_type: VOTER last_known_addr { host: "127.6.235.126" port: 43159 } }
I20260928 02:10:56.726186 9579 raft_consensus.cc:374] T 00000000000000000000000000000000 P f8aeaa8649df40b3ad3523c88291d27f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:10:56.726295 9579 raft_consensus.cc:729] T 00000000000000000000000000000000 P f8aeaa8649df40b3ad3523c88291d27f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f8aeaa8649df40b3ad3523c88291d27f, State: Initialized, Role: FOLLOWER
I20260928 02:10:56.726982 9579 consensus_queue.cc:260] T 00000000000000000000000000000000 P f8aeaa8649df40b3ad3523c88291d27f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f8aeaa8649df40b3ad3523c88291d27f" member_type: VOTER last_known_addr { host: "127.6.235.126" port: 43159 } }
I20260928 02:10:56.727365 9579 raft_consensus.cc:388] T 00000000000000000000000000000000 P f8aeaa8649df40b3ad3523c88291d27f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 02:10:56.727504 9579 raft_consensus.cc:482] T 00000000000000000000000000000000 P f8aeaa8649df40b3ad3523c88291d27f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 02:10:56.727658 9579 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f8aeaa8649df40b3ad3523c88291d27f [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:10:56.729851 9579 raft_consensus.cc:504] T 00000000000000000000000000000000 P f8aeaa8649df40b3ad3523c88291d27f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f8aeaa8649df40b3ad3523c88291d27f" member_type: VOTER last_known_addr { host: "127.6.235.126" port: 43159 } }
I20260928 02:10:56.730291 9579 leader_election.cc:302] T 00000000000000000000000000000000 P f8aeaa8649df40b3ad3523c88291d27f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f8aeaa8649df40b3ad3523c88291d27f; no voters:
I20260928 02:10:56.730928 9579 leader_election.cc:288] T 00000000000000000000000000000000 P f8aeaa8649df40b3ad3523c88291d27f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 02:10:56.731079 9584 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f8aeaa8649df40b3ad3523c88291d27f [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:10:56.732700 9584 raft_consensus.cc:686] T 00000000000000000000000000000000 P f8aeaa8649df40b3ad3523c88291d27f [term 1 LEADER]: Becoming Leader. State: Replica: f8aeaa8649df40b3ad3523c88291d27f, State: Running, Role: LEADER
I20260928 02:10:56.733124 9579 sys_catalog.cc:564] T 00000000000000000000000000000000 P f8aeaa8649df40b3ad3523c88291d27f [sys.catalog]: configured and running, proceeding with master startup.
I20260928 02:10:56.733376 9584 consensus_queue.cc:237] T 00000000000000000000000000000000 P f8aeaa8649df40b3ad3523c88291d27f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f8aeaa8649df40b3ad3523c88291d27f" member_type: VOTER last_known_addr { host: "127.6.235.126" port: 43159 } }
I20260928 02:10:56.736619 9586 sys_catalog.cc:455] T 00000000000000000000000000000000 P f8aeaa8649df40b3ad3523c88291d27f [sys.catalog]: SysCatalogTable state changed. Reason: New leader f8aeaa8649df40b3ad3523c88291d27f. Latest consensus state: current_term: 1 leader_uuid: "f8aeaa8649df40b3ad3523c88291d27f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f8aeaa8649df40b3ad3523c88291d27f" member_type: VOTER last_known_addr { host: "127.6.235.126" port: 43159 } } }
I20260928 02:10:56.736661 9585 sys_catalog.cc:455] T 00000000000000000000000000000000 P f8aeaa8649df40b3ad3523c88291d27f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f8aeaa8649df40b3ad3523c88291d27f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f8aeaa8649df40b3ad3523c88291d27f" member_type: VOTER last_known_addr { host: "127.6.235.126" port: 43159 } } }
I20260928 02:10:56.737053 9586 sys_catalog.cc:458] T 00000000000000000000000000000000 P f8aeaa8649df40b3ad3523c88291d27f [sys.catalog]: This master's current role is: LEADER
I20260928 02:10:56.737053 9585 sys_catalog.cc:458] T 00000000000000000000000000000000 P f8aeaa8649df40b3ad3523c88291d27f [sys.catalog]: This master's current role is: LEADER
I20260928 02:10:56.737977 9594 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 02:10:56.742746 9594 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 02:10:56.751579 9594 catalog_manager.cc:1347] Generated new cluster ID: 6f3ec8e101ee42e68bd5ebc94f8bd543
I20260928 02:10:56.751700 9594 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 02:10:56.766185 9594 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 02:10:56.768213 9594 catalog_manager.cc:1504] Loading token signing keys...
I20260928 02:10:56.787036 9594 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f8aeaa8649df40b3ad3523c88291d27f: Generated new TSK 0
I20260928 02:10:56.788185 9594 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.6.235.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.235.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.235.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790561432251112-7085-0/krb5kdc/kudu_127.6.235.65.keytab.
Entry for principal kudu/127.6.235.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790561432251112-7085-0/krb5kdc/kudu_127.6.235.65.keytab.
I20260928 02:10:56.841331 7085 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTlaUuc/build/asan/bin/kudu
/tmp/dist-test-taskTlaUuc/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790561432251112-7085-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790561432251112-7085-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790561432251112-7085-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790561432251112-7085-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.235.65:0
--local_ip_for_outbound_sockets=127.6.235.65
--webserver_interface=127.6.235.65
--webserver_port=0
--tserver_master_addrs=127.6.235.126:43159
--keytab_file=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790561432251112-7085-0/krb5kdc/kudu_127.6.235.65.keytab
--principal=kudu/127.6.235.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.235.84:39389
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790561432251112-7085-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790561432251112-7085-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790561432251112-7085-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:10:57.177778 9605 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:10:57.178071 9605 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:10:57.178165 9605 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:10:57.185613 9605 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:10:57.185739 9605 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:10:57.185851 9605 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.235.65
I20260928 02:10:57.195744 9605 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.235.84:39389
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790561432251112-7085-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790561432251112-7085-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790561432251112-7085-0/krb5kdc/kudu_127.6.235.65.keytab
--principal=kudu/127.6.235.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.235.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790561432251112-7085-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.6.235.65
--webserver_port=0
--tserver_master_addrs=127.6.235.126:43159
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.235.65
--log_dir=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790561432251112-7085-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 00:51:59 UTC on ccycloud.quasar-axgkwn.root.comops.site
build id 4676464
ASAN enabled
I20260928 02:10:57.197069 9605 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:10:57.198347 9605 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:10:57.211822 9611 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:10:57.211822 9610 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:10:57.212105 9613 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:10:57.212864 9605 server_base.cc:1034] running on GCE node
I20260928 02:10:57.213662 9605 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:10:57.214615 9605 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:10:57.215830 9605 hybrid_clock.cc:648] HybridClock initialized: now 1790561457215768 us; error 59 us; skew 500 ppm
I20260928 02:10:57.216198 9605 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 02:10:57 dist-test-slave-86gv krb5kdc[9499](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561457, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.235.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:10:57.219312 9605 init.cc:377] Logged in from keytab as kudu/127.6.235.65@KRBTEST.COM (short username kudu)
I20260928 02:10:57.220786 9605 webserver.cc:460] Webserver started at http://127.6.235.65:39641/ using document root <none> and password file <none>
I20260928 02:10:57.221514 9605 fs_manager.cc:359] Metadata directory not provided
I20260928 02:10:57.221640 9605 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:10:57.221957 9605 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:10:57.224994 9605 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790561432251112-7085-0/minicluster-data/ts-0/data/instance:
uuid: "428edf051cad482085f7e3706e0d0919"
format_stamp: "Formatted at 2026-09-28 02:10:57 on dist-test-slave-86gv"
server_key: "01f7e4d8f40fd2d8c09065ae2c972f39"
server_key_iv: "0dd46784934ade6455a776d4ac70ae52"
server_key_version: "encryptionkey@0"
I20260928 02:10:57.225835 9605 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790561432251112-7085-0/minicluster-data/ts-0/wal/instance:
uuid: "428edf051cad482085f7e3706e0d0919"
format_stamp: "Formatted at 2026-09-28 02:10:57 on dist-test-slave-86gv"
server_key: "01f7e4d8f40fd2d8c09065ae2c972f39"
server_key_iv: "0dd46784934ade6455a776d4ac70ae52"
server_key_version: "encryptionkey@0"
I20260928 02:10:57.231454 9605 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.004s
I20260928 02:10:57.235855 9620 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:10:57.237694 9605 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 02:10:57.237887 9605 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790561432251112-7085-0/minicluster-data/ts-0/data,/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790561432251112-7085-0/minicluster-data/ts-0/wal
uuid: "428edf051cad482085f7e3706e0d0919"
format_stamp: "Formatted at 2026-09-28 02:10:57 on dist-test-slave-86gv"
server_key: "01f7e4d8f40fd2d8c09065ae2c972f39"
server_key_iv: "0dd46784934ade6455a776d4ac70ae52"
server_key_version: "encryptionkey@0"
I20260928 02:10:57.238092 9605 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790561432251112-7085-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790561432251112-7085-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790561432251112-7085-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:10:57.256199 9605 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:10:57.261407 9605 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:10:57.261714 9605 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:10:57.262738 9605 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:10:57.264194 9605 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:10:57.264274 9605 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:10:57.264423 9605 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:10:57.264478 9605 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:10:57.300772 9605 rpc_server.cc:307] RPC server started. Bound to: 127.6.235.65:46843
I20260928 02:10:57.300834 9732 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.235.65:46843 every 8 connection(s)
I20260928 02:10:57.302150 9605 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790561432251112-7085-0/minicluster-data/ts-0/data/info.pb
I20260928 02:10:57.312175 7085 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTlaUuc/build/asan/bin/kudu as pid 9605
I20260928 02:10:57.312376 7085 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790561432251112-7085-0/minicluster-data/ts-0/wal/instance
I20260928 02:10:57.313227 7085 external_mini_cluster.cc:1428] Setting key 2bddcef2de25f8f2eaba4f8406bd0513
Sep 28 02:10:57 dist-test-slave-86gv krb5kdc[9499](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561457, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.235.65@KRBTEST.COM for kudu/127.6.235.126@KRBTEST.COM
I20260928 02:10:57.323956 9735 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:10:57.305122 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.235.65:46769 (local address 127.6.235.126:43159)
0928 02:10:57.305659 (+ 537us) server_negotiation.cc:206] Beginning negotiation
0928 02:10:57.305664 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 02:10:57.306606 (+ 942us) server_negotiation.cc:407] Connection header received
0928 02:10:57.308536 (+ 1930us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:10:57.308542 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:10:57.308635 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:10:57.308774 (+ 139us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:10:57.310495 (+ 1721us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:10:57.311222 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:10:57.313065 (+ 1843us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:10:57.313537 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:10:57.313678 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:10:57.317679 (+ 4001us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:10:57.317685 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:10:57.317689 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:10:57.317729 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:10:57.320512 (+ 2783us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:10:57.321587 (+ 1075us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:10:57.321592 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:10:57.321597 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:10:57.321717 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:10:57.322182 (+ 465us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:10:57.322188 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:10:57.322191 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:10:57.322466 (+ 275us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:10:57.322618 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0928 02:10:57.323368 (+ 750us) server_negotiation.cc:299] Negotiation successful
0928 02:10:57.323593 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":428,"thread_start_us":326,"threads_started":1}
I20260928 02:10:57.325091 9736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:10:57.305625 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.235.126:43159 (local address 127.6.235.65:46769)
0928 02:10:57.306430 (+ 805us) negotiation.cc:107] Waiting for socket to connect
0928 02:10:57.306480 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0928 02:10:57.308231 (+ 1751us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:10:57.309098 (+ 867us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:10:57.309108 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:10:57.309648 (+ 540us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:10:57.310301 (+ 653us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:10:57.310320 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:10:57.311416 (+ 1096us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:10:57.311420 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:10:57.312832 (+ 1412us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:10:57.312845 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:10:57.313775 (+ 930us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:10:57.313784 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:10:57.314044 (+ 260us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:10:57.314812 (+ 768us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:10:57.314849 (+ 37us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:10:57.317403 (+ 2554us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:10:57.320795 (+ 3392us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:10:57.320810 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:10:57.320837 (+ 27us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:10:57.321433 (+ 596us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:10:57.321854 (+ 421us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:10:57.321859 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:10:57.321864 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:10:57.322058 (+ 194us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:10:57.322603 (+ 545us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:10:57.322612 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:10:57.322983 (+ 371us) client_negotiation.cc:769] Sending connection context
0928 02:10:57.323260 (+ 277us) client_negotiation.cc:240] Negotiation successful
0928 02:10:57.323837 (+ 577us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":490,"thread_start_us":287,"threads_started":1}
I20260928 02:10:57.326725 9733 heartbeater.cc:347] Connected to a master server at 127.6.235.126:43159
I20260928 02:10:57.327085 9733 heartbeater.cc:464] Registering TS with master...
I20260928 02:10:57.328059 9733 heartbeater.cc:511] Master 127.6.235.126:43159 requested a full tablet report, sending...
I20260928 02:10:57.330598 9544 ts_manager.cc:194] Registered new tserver with Master: 428edf051cad482085f7e3706e0d0919 (127.6.235.65:46843)
WARNING: no policy specified for kudu/127.6.235.66@KRBTEST.COM; defaulting to no policy
I20260928 02:10:57.332139 9544 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.235.65@KRBTEST.COM'} at 127.6.235.65:46769
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.235.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.235.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790561432251112-7085-0/krb5kdc/kudu_127.6.235.66.keytab.
Entry for principal kudu/127.6.235.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790561432251112-7085-0/krb5kdc/kudu_127.6.235.66.keytab.
I20260928 02:10:57.361099 7085 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTlaUuc/build/asan/bin/kudu
/tmp/dist-test-taskTlaUuc/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790561432251112-7085-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790561432251112-7085-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790561432251112-7085-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790561432251112-7085-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.235.66:0
--local_ip_for_outbound_sockets=127.6.235.66
--webserver_interface=127.6.235.66
--webserver_port=0
--tserver_master_addrs=127.6.235.126:43159
--keytab_file=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790561432251112-7085-0/krb5kdc/kudu_127.6.235.66.keytab
--principal=kudu/127.6.235.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.235.84:39389
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790561432251112-7085-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790561432251112-7085-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790561432251112-7085-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:10:57.697329 9739 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:10:57.697647 9739 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:10:57.697824 9739 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:10:57.705433 9739 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:10:57.705588 9739 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:10:57.705775 9739 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.235.66
I20260928 02:10:57.715970 9739 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.235.84:39389
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790561432251112-7085-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790561432251112-7085-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790561432251112-7085-0/krb5kdc/kudu_127.6.235.66.keytab
--principal=kudu/127.6.235.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.235.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790561432251112-7085-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.6.235.66
--webserver_port=0
--tserver_master_addrs=127.6.235.126:43159
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.235.66
--log_dir=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790561432251112-7085-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 00:51:59 UTC on ccycloud.quasar-axgkwn.root.comops.site
build id 4676464
ASAN enabled
I20260928 02:10:57.717468 9739 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:10:57.718860 9739 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:10:57.732937 9747 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:10:57.732883 9745 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:10:57.732991 9744 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:10:57.734030 9739 server_base.cc:1034] running on GCE node
I20260928 02:10:57.734728 9739 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:10:57.735788 9739 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:10:57.737030 9739 hybrid_clock.cc:648] HybridClock initialized: now 1790561457736932 us; error 65 us; skew 500 ppm
I20260928 02:10:57.737447 9739 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 02:10:57 dist-test-slave-86gv krb5kdc[9499](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561457, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.235.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:10:57.740733 9739 init.cc:377] Logged in from keytab as kudu/127.6.235.66@KRBTEST.COM (short username kudu)
I20260928 02:10:57.742551 9739 webserver.cc:460] Webserver started at http://127.6.235.66:46463/ using document root <none> and password file <none>
I20260928 02:10:57.743301 9739 fs_manager.cc:359] Metadata directory not provided
I20260928 02:10:57.743434 9739 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:10:57.743786 9739 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:10:57.746914 9739 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790561432251112-7085-0/minicluster-data/ts-1/data/instance:
uuid: "8eaa80c83be34546bf707e44240b3922"
format_stamp: "Formatted at 2026-09-28 02:10:57 on dist-test-slave-86gv"
server_key: "5cc6fe8626e1eaa6ea57e7f2e2d1271f"
server_key_iv: "89bbf80e1958e5e0207d06e12737139a"
server_key_version: "encryptionkey@0"
I20260928 02:10:57.747725 9739 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790561432251112-7085-0/minicluster-data/ts-1/wal/instance:
uuid: "8eaa80c83be34546bf707e44240b3922"
format_stamp: "Formatted at 2026-09-28 02:10:57 on dist-test-slave-86gv"
server_key: "5cc6fe8626e1eaa6ea57e7f2e2d1271f"
server_key_iv: "89bbf80e1958e5e0207d06e12737139a"
server_key_version: "encryptionkey@0"
I20260928 02:10:57.753155 9739 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.005s
I20260928 02:10:57.757095 9754 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:10:57.758615 9739 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260928 02:10:57.758812 9739 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790561432251112-7085-0/minicluster-data/ts-1/data,/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790561432251112-7085-0/minicluster-data/ts-1/wal
uuid: "8eaa80c83be34546bf707e44240b3922"
format_stamp: "Formatted at 2026-09-28 02:10:57 on dist-test-slave-86gv"
server_key: "5cc6fe8626e1eaa6ea57e7f2e2d1271f"
server_key_iv: "89bbf80e1958e5e0207d06e12737139a"
server_key_version: "encryptionkey@0"
I20260928 02:10:57.759051 9739 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790561432251112-7085-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790561432251112-7085-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790561432251112-7085-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:10:57.778060 9739 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:10:57.782837 9739 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:10:57.783094 9739 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:10:57.784085 9739 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:10:57.785482 9739 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:10:57.785554 9739 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:10:57.785640 9739 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:10:57.785683 9739 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:10:57.818527 9739 rpc_server.cc:307] RPC server started. Bound to: 127.6.235.66:40835
I20260928 02:10:57.818572 9866 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.235.66:40835 every 8 connection(s)
I20260928 02:10:57.819833 9739 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790561432251112-7085-0/minicluster-data/ts-1/data/info.pb
I20260928 02:10:57.822994 7085 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTlaUuc/build/asan/bin/kudu as pid 9739
I20260928 02:10:57.823185 7085 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790561432251112-7085-0/minicluster-data/ts-1/wal/instance
I20260928 02:10:57.823669 7085 external_mini_cluster.cc:1428] Setting key 76ecd4ac0ccbc08cc07dcdd8c8fb0d35
Sep 28 02:10:57 dist-test-slave-86gv krb5kdc[9499](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561457, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.235.66@KRBTEST.COM for kudu/127.6.235.126@KRBTEST.COM
I20260928 02:10:57.841655 9735 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:10:57.822430 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.235.66:54733 (local address 127.6.235.126:43159)
0928 02:10:57.822641 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0928 02:10:57.822648 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 02:10:57.824037 (+ 1389us) server_negotiation.cc:407] Connection header received
0928 02:10:57.825933 (+ 1896us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:10:57.825939 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:10:57.826018 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:10:57.826121 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:10:57.827635 (+ 1514us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:10:57.828766 (+ 1131us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:10:57.830617 (+ 1851us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:10:57.831193 (+ 576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:10:57.831293 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:10:57.835604 (+ 4311us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:10:57.835611 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:10:57.835616 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:10:57.835661 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:10:57.838386 (+ 2725us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:10:57.839296 (+ 910us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:10:57.839300 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:10:57.839303 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:10:57.839436 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:10:57.839890 (+ 454us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:10:57.839900 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:10:57.839903 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:10:57.840162 (+ 259us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:10:57.840313 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0928 02:10:57.841082 (+ 769us) server_negotiation.cc:299] Negotiation successful
0928 02:10:57.841307 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
WARNING: no policy specified for kudu/127.6.235.67@KRBTEST.COM; defaulting to no policy
I20260928 02:10:57.842857 9869 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:10:57.822961 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.235.126:43159 (local address 127.6.235.66:54733)
0928 02:10:57.823844 (+ 883us) negotiation.cc:107] Waiting for socket to connect
0928 02:10:57.823889 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0928 02:10:57.825672 (+ 1783us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:10:57.826372 (+ 700us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:10:57.826380 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:10:57.826884 (+ 504us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:10:57.827462 (+ 578us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:10:57.827479 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:10:57.829022 (+ 1543us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:10:57.829026 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:10:57.830472 (+ 1446us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:10:57.830483 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:10:57.831428 (+ 945us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:10:57.831447 (+ 19us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:10:57.831813 (+ 366us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:10:57.832597 (+ 784us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:10:57.832631 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:10:57.835359 (+ 2728us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:10:57.838561 (+ 3202us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:10:57.838572 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:10:57.838592 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:10:57.839155 (+ 563us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:10:57.839550 (+ 395us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:10:57.839555 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:10:57.839559 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:10:57.839767 (+ 208us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:10:57.840298 (+ 531us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:10:57.840306 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:10:57.840673 (+ 367us) client_negotiation.cc:769] Sending connection context
0928 02:10:57.840983 (+ 310us) client_negotiation.cc:240] Negotiation successful
0928 02:10:57.841536 (+ 553us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":595,"thread_start_us":372,"threads_started":1}
I20260928 02:10:57.844372 9867 heartbeater.cc:347] Connected to a master server at 127.6.235.126:43159
I20260928 02:10:57.844703 9867 heartbeater.cc:464] Registering TS with master...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.235.67@KRBTEST.COM" created.
I20260928 02:10:57.845721 9867 heartbeater.cc:511] Master 127.6.235.126:43159 requested a full tablet report, sending...
I20260928 02:10:57.847357 9544 ts_manager.cc:194] Registered new tserver with Master: 8eaa80c83be34546bf707e44240b3922 (127.6.235.66:40835)
I20260928 02:10:57.848179 9544 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.235.66@KRBTEST.COM'} at 127.6.235.66:54733
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.235.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790561432251112-7085-0/krb5kdc/kudu_127.6.235.67.keytab.
Entry for principal kudu/127.6.235.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790561432251112-7085-0/krb5kdc/kudu_127.6.235.67.keytab.
I20260928 02:10:57.871486 7085 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTlaUuc/build/asan/bin/kudu
/tmp/dist-test-taskTlaUuc/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790561432251112-7085-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790561432251112-7085-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790561432251112-7085-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790561432251112-7085-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.235.67:0
--local_ip_for_outbound_sockets=127.6.235.67
--webserver_interface=127.6.235.67
--webserver_port=0
--tserver_master_addrs=127.6.235.126:43159
--keytab_file=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790561432251112-7085-0/krb5kdc/kudu_127.6.235.67.keytab
--principal=kudu/127.6.235.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.235.84:39389
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790561432251112-7085-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790561432251112-7085-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790561432251112-7085-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:10:58.210742 9872 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:10:58.211054 9872 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:10:58.211210 9872 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:10:58.218577 9872 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:10:58.218696 9872 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:10:58.218906 9872 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.235.67
I20260928 02:10:58.229027 9872 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.235.84:39389
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790561432251112-7085-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790561432251112-7085-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790561432251112-7085-0/krb5kdc/kudu_127.6.235.67.keytab
--principal=kudu/127.6.235.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.235.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790561432251112-7085-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.6.235.67
--webserver_port=0
--tserver_master_addrs=127.6.235.126:43159
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.235.67
--log_dir=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790561432251112-7085-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 00:51:59 UTC on ccycloud.quasar-axgkwn.root.comops.site
build id 4676464
ASAN enabled
I20260928 02:10:58.230402 9872 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:10:58.231783 9872 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:10:58.245596 9877 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:10:58.245546 9880 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:10:58.245839 9878 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:10:58.246905 9872 server_base.cc:1034] running on GCE node
I20260928 02:10:58.247563 9872 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:10:58.248543 9872 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:10:58.249780 9872 hybrid_clock.cc:648] HybridClock initialized: now 1790561458249707 us; error 61 us; skew 500 ppm
I20260928 02:10:58.250136 9872 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 02:10:58 dist-test-slave-86gv krb5kdc[9499](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561458, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.235.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:10:58.253767 9872 init.cc:377] Logged in from keytab as kudu/127.6.235.67@KRBTEST.COM (short username kudu)
I20260928 02:10:58.255591 9872 webserver.cc:460] Webserver started at http://127.6.235.67:37313/ using document root <none> and password file <none>
I20260928 02:10:58.256414 9872 fs_manager.cc:359] Metadata directory not provided
I20260928 02:10:58.256546 9872 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:10:58.256880 9872 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:10:58.259893 9872 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790561432251112-7085-0/minicluster-data/ts-2/data/instance:
uuid: "332d02f7258a4bb9a50d4a33dfbf032c"
format_stamp: "Formatted at 2026-09-28 02:10:58 on dist-test-slave-86gv"
server_key: "a2ecd410dfe3ceccf4d11a5dbfff0646"
server_key_iv: "d59d7c83e1ff2d3315806f57e2fd1328"
server_key_version: "encryptionkey@0"
I20260928 02:10:58.260741 9872 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790561432251112-7085-0/minicluster-data/ts-2/wal/instance:
uuid: "332d02f7258a4bb9a50d4a33dfbf032c"
format_stamp: "Formatted at 2026-09-28 02:10:58 on dist-test-slave-86gv"
server_key: "a2ecd410dfe3ceccf4d11a5dbfff0646"
server_key_iv: "d59d7c83e1ff2d3315806f57e2fd1328"
server_key_version: "encryptionkey@0"
I20260928 02:10:58.266472 9872 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260928 02:10:58.270390 9887 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:10:58.271862 9872 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260928 02:10:58.272081 9872 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790561432251112-7085-0/minicluster-data/ts-2/data,/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790561432251112-7085-0/minicluster-data/ts-2/wal
uuid: "332d02f7258a4bb9a50d4a33dfbf032c"
format_stamp: "Formatted at 2026-09-28 02:10:58 on dist-test-slave-86gv"
server_key: "a2ecd410dfe3ceccf4d11a5dbfff0646"
server_key_iv: "d59d7c83e1ff2d3315806f57e2fd1328"
server_key_version: "encryptionkey@0"
I20260928 02:10:58.272439 9872 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790561432251112-7085-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790561432251112-7085-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790561432251112-7085-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:10:58.286659 9872 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:10:58.291260 9872 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:10:58.291570 9872 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:10:58.292623 9872 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:10:58.294059 9872 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:10:58.294130 9872 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:10:58.294281 9872 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:10:58.294330 9872 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:10:58.329250 9872 rpc_server.cc:307] RPC server started. Bound to: 127.6.235.67:33385
I20260928 02:10:58.329252 9999 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.235.67:33385 every 8 connection(s)
I20260928 02:10:58.330960 9872 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790561432251112-7085-0/minicluster-data/ts-2/data/info.pb
I20260928 02:10:58.331759 7085 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTlaUuc/build/asan/bin/kudu as pid 9872
I20260928 02:10:58.331939 7085 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790561432251112-7085-0/minicluster-data/ts-2/wal/instance
I20260928 02:10:58.332382 7085 external_mini_cluster.cc:1428] Setting key 88c6fe3af5c9e4e6defb307795d52c6c
I20260928 02:10:58.335809 9733 heartbeater.cc:503] Master 127.6.235.126:43159 was elected leader, sending a full tablet report...
Sep 28 02:10:58 dist-test-slave-86gv krb5kdc[9499](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561458, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.235.67@KRBTEST.COM for kudu/127.6.235.126@KRBTEST.COM
I20260928 02:10:58.355476 9735 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:10:58.334837 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.235.67:35147 (local address 127.6.235.126:43159)
0928 02:10:58.335061 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0928 02:10:58.335068 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 02:10:58.336772 (+ 1704us) server_negotiation.cc:407] Connection header received
0928 02:10:58.338845 (+ 2073us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:10:58.338851 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:10:58.338939 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:10:58.339064 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:10:58.340655 (+ 1591us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:10:58.341561 (+ 906us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:10:58.343264 (+ 1703us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:10:58.343849 (+ 585us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:10:58.343985 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:10:58.347746 (+ 3761us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:10:58.347752 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:10:58.347756 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:10:58.347797 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:10:58.352086 (+ 4289us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:10:58.353072 (+ 986us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:10:58.353078 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:10:58.353081 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:10:58.353221 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:10:58.353693 (+ 472us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:10:58.353698 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:10:58.353702 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:10:58.353984 (+ 282us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:10:58.354139 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0928 02:10:58.354886 (+ 747us) server_negotiation.cc:299] Negotiation successful
0928 02:10:58.355101 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87}
I20260928 02:10:58.356496 10002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:10:58.335576 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.235.126:43159 (local address 127.6.235.67:35147)
0928 02:10:58.336585 (+ 1009us) negotiation.cc:107] Waiting for socket to connect
0928 02:10:58.336646 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0928 02:10:58.338499 (+ 1853us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:10:58.339328 (+ 829us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:10:58.339337 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:10:58.339883 (+ 546us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:10:58.340477 (+ 594us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:10:58.340495 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:10:58.341709 (+ 1214us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:10:58.341714 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:10:58.343091 (+ 1377us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:10:58.343103 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:10:58.344090 (+ 987us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:10:58.344096 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:10:58.344344 (+ 248us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:10:58.345127 (+ 783us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:10:58.345150 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:10:58.347509 (+ 2359us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:10:58.352249 (+ 4740us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:10:58.352259 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:10:58.352279 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:10:58.352871 (+ 592us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:10:58.353349 (+ 478us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:10:58.353354 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:10:58.353358 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:10:58.353568 (+ 210us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:10:58.354116 (+ 548us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:10:58.354125 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:10:58.354503 (+ 378us) client_negotiation.cc:769] Sending connection context
0928 02:10:58.354788 (+ 285us) client_negotiation.cc:240] Negotiation successful
0928 02:10:58.355364 (+ 576us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":705,"thread_start_us":423,"threads_started":1}
I20260928 02:10:58.357939 10000 heartbeater.cc:347] Connected to a master server at 127.6.235.126:43159
I20260928 02:10:58.358282 10000 heartbeater.cc:464] Registering TS with master...
I20260928 02:10:58.359051 10000 heartbeater.cc:511] Master 127.6.235.126:43159 requested a full tablet report, sending...
I20260928 02:10:58.360952 9544 ts_manager.cc:194] Registered new tserver with Master: 332d02f7258a4bb9a50d4a33dfbf032c (127.6.235.67:33385)
I20260928 02:10:58.361771 9544 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.235.67@KRBTEST.COM'} at 127.6.235.67:35147
I20260928 02:10:58.372872 7085 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260928 02:10:58.406414 9735 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:10:58.403607 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57650 (local address 127.6.235.126:43159)
0928 02:10:58.403882 (+ 275us) server_negotiation.cc:206] Beginning negotiation
0928 02:10:58.403888 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 02:10:58.404213 (+ 325us) server_negotiation.cc:407] Connection header received
0928 02:10:58.404385 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:10:58.404389 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:10:58.404492 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:10:58.404583 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:10:58.406149 (+ 1566us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57650: BlockingRecv error: recv got EOF from 127.0.0.1:57650 (error 108)
Metrics: {"server-negotiator.queue_time_us":71}
I20260928 02:10:58.410274 7085 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskTlaUuc/build/asan/bin/kudu pid 9605
W20260928 02:10:58.413609 7085 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskTlaUuc/build/asan/bin/kudu pid 9605: Not authorized: Client connection negotiation failed: client connection to 127.6.235.65:46843: server requires authentication, but client does not have Kerberos credentials available
I20260928 02:10:58.413844 7085 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTlaUuc/build/asan/bin/kudu with pid 9605
I20260928 02:10:58.432308 7085 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskTlaUuc/build/asan/bin/kudu pid 9739
W20260928 02:10:58.435652 7085 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskTlaUuc/build/asan/bin/kudu pid 9739: Not authorized: Client connection negotiation failed: client connection to 127.6.235.66:40835: server requires authentication, but client does not have Kerberos credentials available
I20260928 02:10:58.435920 7085 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTlaUuc/build/asan/bin/kudu with pid 9739
I20260928 02:10:58.452536 7085 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskTlaUuc/build/asan/bin/kudu pid 9872
W20260928 02:10:58.455595 7085 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskTlaUuc/build/asan/bin/kudu pid 9872: Not authorized: Client connection negotiation failed: client connection to 127.6.235.67:33385: server requires authentication, but client does not have Kerberos credentials available
I20260928 02:10:58.455770 7085 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTlaUuc/build/asan/bin/kudu with pid 9872
I20260928 02:10:58.471619 7085 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskTlaUuc/build/asan/bin/kudu pid 9512
I20260928 02:10:58.472851 9574 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 02:10:58.575486 7085 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTlaUuc/build/asan/bin/kudu with pid 9512
2026-09-28T02:10:58Z chronyd exiting
[ OK ] SecurityITest.TestNoKerberosCredentials (3905 ms)
[ RUN ] SecurityITest.SaslPlainFallback
Loading random data
Initializing database '/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790561432251112-7085-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 02:10:58 dist-test-slave-86gv krb5kdc[10017](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 02:10:58 dist-test-slave-86gv krb5kdc[10017](info): set up 2 sockets
Sep 28 02:10:58 dist-test-slave-86gv krb5kdc[10017](info): commencing operation
krb5kdc: starting...
W20260928 02:10:59.705874 7085 mini_kdc.cc:121] Time spent starting KDC: real 1.087s user 0.001s sys 0.017s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790561432251112-7085-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790561432251112-7085-0/krb5kdc/test-user.keytab.
Sep 28 02:10:59 dist-test-slave-86gv krb5kdc[10017](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561459, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T02:10:59Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T02:10:59Z Disabled control of system clock
WARNING: no policy specified for kudu/127.6.235.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.235.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.235.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790561432251112-7085-0/krb5kdc/kudu_127.6.235.126.keytab.
Entry for principal kudu/127.6.235.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790561432251112-7085-0/krb5kdc/kudu_127.6.235.126.keytab.
I20260928 02:10:59.909366 7085 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTlaUuc/build/asan/bin/kudu
/tmp/dist-test-taskTlaUuc/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790561432251112-7085-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790561432251112-7085-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790561432251112-7085-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790561432251112-7085-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.235.126:36515
--webserver_interface=127.6.235.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790561432251112-7085-0/krb5kdc/kudu_127.6.235.126.keytab
--principal=kudu/127.6.235.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.235.84:38621
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.235.126:36515
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc-authentication=optional
--user-acl=* with env {KRB5CCNAME=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790561432251112-7085-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790561432251112-7085-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790561432251112-7085-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:11:00.241258 10030 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:11:00.241624 10030 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:11:00.241787 10030 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:11:00.249148 10030 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 02:11:00.249312 10030 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:11:00.249429 10030 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:11:00.249524 10030 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 02:11:00.249608 10030 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 02:11:00.259734 10030 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.235.84:38621
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790561432251112-7085-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790561432251112-7085-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.235.126:36515
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790561432251112-7085-0/krb5kdc/kudu_127.6.235.126.keytab
--principal=kudu/127.6.235.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.235.126:36515
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790561432251112-7085-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.235.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790561432251112-7085-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 00:51:59 UTC on ccycloud.quasar-axgkwn.root.comops.site
build id 4676464
ASAN enabled
I20260928 02:11:00.261145 10030 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:11:00.262507 10030 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:11:00.274725 10038 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:11:00.274855 10035 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:11:00.274791 10036 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:11:00.276134 10030 server_base.cc:1034] running on GCE node
I20260928 02:11:00.277086 10030 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:11:00.278648 10030 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:11:00.279894 10030 hybrid_clock.cc:648] HybridClock initialized: now 1790561460279815 us; error 59 us; skew 500 ppm
I20260928 02:11:00.280309 10030 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 02:11:00 dist-test-slave-86gv krb5kdc[10017](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561460, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.235.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:11:00.283864 10030 init.cc:377] Logged in from keytab as kudu/127.6.235.126@KRBTEST.COM (short username kudu)
I20260928 02:11:00.285494 10030 webserver.cc:460] Webserver started at http://127.6.235.126:38701/ using document root <none> and password file <none>
I20260928 02:11:00.286212 10030 fs_manager.cc:359] Metadata directory not provided
I20260928 02:11:00.286327 10030 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:11:00.286636 10030 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:11:00.289970 10030 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790561432251112-7085-0/minicluster-data/master-0/data/instance:
uuid: "a694099c88904940bcb4a566fe4f621a"
format_stamp: "Formatted at 2026-09-28 02:11:00 on dist-test-slave-86gv"
server_key: "1ed3a9e7097e354b27cb769b9a0b6abb"
server_key_iv: "a1450eb6947754695b373f78b46ac3a8"
server_key_version: "encryptionkey@0"
I20260928 02:11:00.290798 10030 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790561432251112-7085-0/minicluster-data/master-0/wal/instance:
uuid: "a694099c88904940bcb4a566fe4f621a"
format_stamp: "Formatted at 2026-09-28 02:11:00 on dist-test-slave-86gv"
server_key: "1ed3a9e7097e354b27cb769b9a0b6abb"
server_key_iv: "a1450eb6947754695b373f78b46ac3a8"
server_key_version: "encryptionkey@0"
I20260928 02:11:00.296351 10030 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20260928 02:11:00.300134 10045 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:11:00.301620 10030 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260928 02:11:00.301797 10030 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790561432251112-7085-0/minicluster-data/master-0/data,/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790561432251112-7085-0/minicluster-data/master-0/wal
uuid: "a694099c88904940bcb4a566fe4f621a"
format_stamp: "Formatted at 2026-09-28 02:11:00 on dist-test-slave-86gv"
server_key: "1ed3a9e7097e354b27cb769b9a0b6abb"
server_key_iv: "a1450eb6947754695b373f78b46ac3a8"
server_key_version: "encryptionkey@0"
I20260928 02:11:00.302019 10030 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790561432251112-7085-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790561432251112-7085-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790561432251112-7085-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:11:00.326251 10030 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:11:00.330933 10030 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:11:00.331280 10030 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:11:00.350502 10030 rpc_server.cc:307] RPC server started. Bound to: 127.6.235.126:36515
I20260928 02:11:00.350520 10096 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.235.126:36515 every 8 connection(s)
I20260928 02:11:00.352010 10030 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790561432251112-7085-0/minicluster-data/master-0/data/info.pb
I20260928 02:11:00.356366 10097 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:11:00.359277 7085 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTlaUuc/build/asan/bin/kudu as pid 10030
I20260928 02:11:00.359548 7085 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790561432251112-7085-0/minicluster-data/master-0/wal/instance
I20260928 02:11:00.360296 7085 external_mini_cluster.cc:1428] Setting key 34f983cd23541f610de15cb1b0214091
I20260928 02:11:00.371337 10097 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a694099c88904940bcb4a566fe4f621a: Bootstrap starting.
I20260928 02:11:00.375209 10097 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a694099c88904940bcb4a566fe4f621a: Neither blocks nor log segments found. Creating new log.
I20260928 02:11:00.376256 10097 log.cc:824] T 00000000000000000000000000000000 P a694099c88904940bcb4a566fe4f621a: Log is configured to *not* fsync() on all Append() calls
Sep 28 02:11:00 dist-test-slave-86gv krb5kdc[10017](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561459, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.235.126@KRBTEST.COM
I20260928 02:11:00.379215 10097 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a694099c88904940bcb4a566fe4f621a: No bootstrap required, opened a new log
I20260928 02:11:00.384980 10100 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:11:00.362928 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54190 (local address 127.6.235.126:36515)
0928 02:11:00.363887 (+ 959us) server_negotiation.cc:206] Beginning negotiation
0928 02:11:00.363904 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0928 02:11:00.363959 (+ 55us) server_negotiation.cc:407] Connection header received
0928 02:11:00.366293 (+ 2334us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:11:00.366338 (+ 45us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:11:00.366897 (+ 559us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:11:00.367411 (+ 514us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:11:00.368780 (+ 1369us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:11:00.370790 (+ 2010us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:11:00.372202 (+ 1412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:11:00.373339 (+ 1137us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:11:00.373543 (+ 204us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:11:00.377615 (+ 4072us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:11:00.377623 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:11:00.377640 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:11:00.377677 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:11:00.380740 (+ 3063us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:11:00.381497 (+ 757us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:11:00.381504 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:11:00.381511 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:11:00.381682 (+ 171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:11:00.382066 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:11:00.382071 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:11:00.382076 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:11:00.382661 (+ 585us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:11:00.383015 (+ 354us) server_negotiation.cc:1035] Waiting for connection context
0928 02:11:00.383434 (+ 419us) server_negotiation.cc:299] Negotiation successful
0928 02:11:00.383856 (+ 422us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":691,"thread_start_us":416,"threads_started":1}
I20260928 02:11:00.387583 10097 raft_consensus.cc:348] T 00000000000000000000000000000000 P a694099c88904940bcb4a566fe4f621a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a694099c88904940bcb4a566fe4f621a" member_type: VOTER last_known_addr { host: "127.6.235.126" port: 36515 } }
I20260928 02:11:00.387984 10097 raft_consensus.cc:374] T 00000000000000000000000000000000 P a694099c88904940bcb4a566fe4f621a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:11:00.388103 10097 raft_consensus.cc:729] T 00000000000000000000000000000000 P a694099c88904940bcb4a566fe4f621a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a694099c88904940bcb4a566fe4f621a, State: Initialized, Role: FOLLOWER
I20260928 02:11:00.388715 10097 consensus_queue.cc:260] T 00000000000000000000000000000000 P a694099c88904940bcb4a566fe4f621a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a694099c88904940bcb4a566fe4f621a" member_type: VOTER last_known_addr { host: "127.6.235.126" port: 36515 } }
I20260928 02:11:00.389053 10097 raft_consensus.cc:388] T 00000000000000000000000000000000 P a694099c88904940bcb4a566fe4f621a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 02:11:00.389132 7085 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 02:11:00.389189 10097 raft_consensus.cc:482] T 00000000000000000000000000000000 P a694099c88904940bcb4a566fe4f621a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 02:11:00.389380 10097 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a694099c88904940bcb4a566fe4f621a [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:11:00.391534 10097 raft_consensus.cc:504] T 00000000000000000000000000000000 P a694099c88904940bcb4a566fe4f621a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a694099c88904940bcb4a566fe4f621a" member_type: VOTER last_known_addr { host: "127.6.235.126" port: 36515 } }
I20260928 02:11:00.392068 10097 leader_election.cc:302] T 00000000000000000000000000000000 P a694099c88904940bcb4a566fe4f621a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a694099c88904940bcb4a566fe4f621a; no voters:
I20260928 02:11:00.392701 10097 leader_election.cc:288] T 00000000000000000000000000000000 P a694099c88904940bcb4a566fe4f621a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 02:11:00.392853 10102 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a694099c88904940bcb4a566fe4f621a [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:11:00.394143 10102 raft_consensus.cc:686] T 00000000000000000000000000000000 P a694099c88904940bcb4a566fe4f621a [term 1 LEADER]: Becoming Leader. State: Replica: a694099c88904940bcb4a566fe4f621a, State: Running, Role: LEADER
I20260928 02:11:00.394721 10102 consensus_queue.cc:237] T 00000000000000000000000000000000 P a694099c88904940bcb4a566fe4f621a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a694099c88904940bcb4a566fe4f621a" member_type: VOTER last_known_addr { host: "127.6.235.126" port: 36515 } }
I20260928 02:11:00.395072 10097 sys_catalog.cc:564] T 00000000000000000000000000000000 P a694099c88904940bcb4a566fe4f621a [sys.catalog]: configured and running, proceeding with master startup.
I20260928 02:11:00.397921 10104 sys_catalog.cc:455] T 00000000000000000000000000000000 P a694099c88904940bcb4a566fe4f621a [sys.catalog]: SysCatalogTable state changed. Reason: New leader a694099c88904940bcb4a566fe4f621a. Latest consensus state: current_term: 1 leader_uuid: "a694099c88904940bcb4a566fe4f621a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a694099c88904940bcb4a566fe4f621a" member_type: VOTER last_known_addr { host: "127.6.235.126" port: 36515 } } }
I20260928 02:11:00.398286 10104 sys_catalog.cc:458] T 00000000000000000000000000000000 P a694099c88904940bcb4a566fe4f621a [sys.catalog]: This master's current role is: LEADER
I20260928 02:11:00.398819 10103 sys_catalog.cc:455] T 00000000000000000000000000000000 P a694099c88904940bcb4a566fe4f621a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a694099c88904940bcb4a566fe4f621a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a694099c88904940bcb4a566fe4f621a" member_type: VOTER last_known_addr { host: "127.6.235.126" port: 36515 } } }
I20260928 02:11:00.399175 10103 sys_catalog.cc:458] T 00000000000000000000000000000000 P a694099c88904940bcb4a566fe4f621a [sys.catalog]: This master's current role is: LEADER
I20260928 02:11:00.402091 10113 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 02:11:00.407158 10113 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 02:11:00.429023 10113 catalog_manager.cc:1347] Generated new cluster ID: 7fac45d13d8541b6b652220ded29db70
I20260928 02:11:00.429188 10113 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 02:11:00.446092 10113 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 02:11:00.448576 10113 catalog_manager.cc:1504] Loading token signing keys...
I20260928 02:11:00.452929 10123 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20260928 02:11:00.458133 10113 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a694099c88904940bcb4a566fe4f621a: Generated new TSK 0
I20260928 02:11:00.459070 10113 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 02:11:00.464859 10100 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:11:00.455586 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54204 (local address 127.6.235.126:36515)
0928 02:11:00.455760 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0928 02:11:00.455767 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 02:11:00.456342 (+ 575us) server_negotiation.cc:407] Connection header received
0928 02:11:00.456539 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:11:00.456544 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:11:00.456623 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:11:00.456734 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:11:00.458120 (+ 1386us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:11:00.459315 (+ 1195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:11:00.461326 (+ 2011us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:11:00.461750 (+ 424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:11:00.461855 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:11:00.463054 (+ 1199us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:11:00.463059 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:11:00.463062 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 02:11:00.463926 (+ 864us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:11:00.463985 (+ 59us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 02:11:00.464009 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:11:00.464110 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0928 02:11:00.464315 (+ 205us) server_negotiation.cc:299] Negotiation successful
0928 02:11:00.464423 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260928 02:11:00.472330 7085 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskTlaUuc/build/asan/bin/kudu pid 10030
I20260928 02:11:00.473611 10092 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 02:11:00.562041 7085 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTlaUuc/build/asan/bin/kudu with pid 10030
2026-09-28T02:11:00Z chronyd exiting
[ OK ] SecurityITest.SaslPlainFallback (1980 ms)
[ RUN ] SecurityITest.TestUnauthorizedClientKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790561432251112-7085-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 02:11:00 dist-test-slave-86gv krb5kdc[10131](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 02:11:00 dist-test-slave-86gv krb5kdc[10131](info): set up 2 sockets
Sep 28 02:11:00 dist-test-slave-86gv krb5kdc[10131](info): commencing operation
krb5kdc: starting...
W20260928 02:11:01.649685 7085 mini_kdc.cc:121] Time spent starting KDC: real 1.050s user 0.002s sys 0.017s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790561432251112-7085-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790561432251112-7085-0/krb5kdc/test-user.keytab.
Sep 28 02:11:01 dist-test-slave-86gv krb5kdc[10131](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561461, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T02:11:01Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T02:11:01Z Disabled control of system clock
WARNING: no policy specified for kudu/127.6.235.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.235.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.235.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790561432251112-7085-0/krb5kdc/kudu_127.6.235.126.keytab.
Entry for principal kudu/127.6.235.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790561432251112-7085-0/krb5kdc/kudu_127.6.235.126.keytab.
I20260928 02:11:01.857522 7085 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTlaUuc/build/asan/bin/kudu
/tmp/dist-test-taskTlaUuc/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790561432251112-7085-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790561432251112-7085-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790561432251112-7085-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790561432251112-7085-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.235.126:40897
--webserver_interface=127.6.235.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790561432251112-7085-0/krb5kdc/kudu_127.6.235.126.keytab
--principal=kudu/127.6.235.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.235.84:36177
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.235.126:40897
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790561432251112-7085-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790561432251112-7085-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790561432251112-7085-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:11:02.192830 10144 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:11:02.193264 10144 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:11:02.193428 10144 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:11:02.201068 10144 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 02:11:02.201186 10144 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:11:02.201298 10144 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:11:02.201375 10144 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 02:11:02.201411 10144 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 02:11:02.211999 10144 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.235.84:36177
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790561432251112-7085-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790561432251112-7085-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.235.126:40897
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790561432251112-7085-0/krb5kdc/kudu_127.6.235.126.keytab
--principal=kudu/127.6.235.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.235.126:40897
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790561432251112-7085-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.235.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790561432251112-7085-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 00:51:59 UTC on ccycloud.quasar-axgkwn.root.comops.site
build id 4676464
ASAN enabled
I20260928 02:11:02.213642 10144 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:11:02.215016 10144 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:11:02.227715 10152 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:11:02.227622 10149 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:11:02.227644 10150 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:11:02.228489 10144 server_base.cc:1034] running on GCE node
I20260928 02:11:02.229447 10144 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:11:02.230957 10144 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:11:02.232198 10144 hybrid_clock.cc:648] HybridClock initialized: now 1790561462232140 us; error 39 us; skew 500 ppm
I20260928 02:11:02.232782 10144 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 02:11:02 dist-test-slave-86gv krb5kdc[10131](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561462, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.235.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:11:02.236418 10144 init.cc:377] Logged in from keytab as kudu/127.6.235.126@KRBTEST.COM (short username kudu)
I20260928 02:11:02.238102 10144 webserver.cc:460] Webserver started at http://127.6.235.126:43433/ using document root <none> and password file <none>
I20260928 02:11:02.238867 10144 fs_manager.cc:359] Metadata directory not provided
I20260928 02:11:02.239032 10144 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:11:02.239408 10144 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:11:02.242935 10144 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790561432251112-7085-0/minicluster-data/master-0/data/instance:
uuid: "16b947b26d5b42b5b3948a7decdf7a00"
format_stamp: "Formatted at 2026-09-28 02:11:02 on dist-test-slave-86gv"
server_key: "6222417edddea3350e5cf07ef9c0dec4"
server_key_iv: "90eb7f01871c4773fc7c2034315f40d4"
server_key_version: "encryptionkey@0"
I20260928 02:11:02.243908 10144 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790561432251112-7085-0/minicluster-data/master-0/wal/instance:
uuid: "16b947b26d5b42b5b3948a7decdf7a00"
format_stamp: "Formatted at 2026-09-28 02:11:02 on dist-test-slave-86gv"
server_key: "6222417edddea3350e5cf07ef9c0dec4"
server_key_iv: "90eb7f01871c4773fc7c2034315f40d4"
server_key_version: "encryptionkey@0"
I20260928 02:11:02.250061 10144 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.001s
I20260928 02:11:02.254010 10159 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:11:02.255360 10144 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260928 02:11:02.255565 10144 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790561432251112-7085-0/minicluster-data/master-0/data,/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790561432251112-7085-0/minicluster-data/master-0/wal
uuid: "16b947b26d5b42b5b3948a7decdf7a00"
format_stamp: "Formatted at 2026-09-28 02:11:02 on dist-test-slave-86gv"
server_key: "6222417edddea3350e5cf07ef9c0dec4"
server_key_iv: "90eb7f01871c4773fc7c2034315f40d4"
server_key_version: "encryptionkey@0"
I20260928 02:11:02.255770 10144 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790561432251112-7085-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790561432251112-7085-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790561432251112-7085-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:11:02.276728 10144 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:11:02.280998 10144 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:11:02.281308 10144 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:11:02.300439 10144 rpc_server.cc:307] RPC server started. Bound to: 127.6.235.126:40897
I20260928 02:11:02.300468 10210 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.235.126:40897 every 8 connection(s)
I20260928 02:11:02.301872 10144 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790561432251112-7085-0/minicluster-data/master-0/data/info.pb
I20260928 02:11:02.305725 10211 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:11:02.307626 7085 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTlaUuc/build/asan/bin/kudu as pid 10144
I20260928 02:11:02.307863 7085 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790561432251112-7085-0/minicluster-data/master-0/wal/instance
I20260928 02:11:02.308490 7085 external_mini_cluster.cc:1428] Setting key 48086b54f7f4891f2476da54d3eaf4ee
I20260928 02:11:02.317957 10211 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 16b947b26d5b42b5b3948a7decdf7a00: Bootstrap starting.
I20260928 02:11:02.321858 10211 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 16b947b26d5b42b5b3948a7decdf7a00: Neither blocks nor log segments found. Creating new log.
I20260928 02:11:02.323252 10211 log.cc:824] T 00000000000000000000000000000000 P 16b947b26d5b42b5b3948a7decdf7a00: Log is configured to *not* fsync() on all Append() calls
Sep 28 02:11:02 dist-test-slave-86gv krb5kdc[10131](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561461, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.235.126@KRBTEST.COM
I20260928 02:11:02.326599 10211 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 16b947b26d5b42b5b3948a7decdf7a00: No bootstrap required, opened a new log
I20260928 02:11:02.332864 10214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:11:02.310788 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58922 (local address 127.6.235.126:40897)
0928 02:11:02.311584 (+ 796us) server_negotiation.cc:206] Beginning negotiation
0928 02:11:02.311597 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0928 02:11:02.311637 (+ 40us) server_negotiation.cc:407] Connection header received
0928 02:11:02.313402 (+ 1765us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:11:02.313426 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:11:02.313828 (+ 402us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:11:02.314171 (+ 343us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:11:02.315935 (+ 1764us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:11:02.317412 (+ 1477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:11:02.319147 (+ 1735us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:11:02.319918 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:11:02.320077 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:11:02.325134 (+ 5057us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:11:02.325142 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:11:02.325160 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:11:02.325202 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:11:02.328365 (+ 3163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:11:02.329217 (+ 852us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:11:02.329224 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:11:02.329232 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:11:02.329409 (+ 177us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:11:02.329807 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:11:02.329813 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:11:02.329817 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:11:02.330411 (+ 594us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:11:02.330794 (+ 383us) server_negotiation.cc:1035] Waiting for connection context
0928 02:11:02.331183 (+ 389us) server_negotiation.cc:299] Negotiation successful
0928 02:11:02.331643 (+ 460us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":586,"thread_start_us":375,"threads_started":1}
I20260928 02:11:02.334892 10211 raft_consensus.cc:348] T 00000000000000000000000000000000 P 16b947b26d5b42b5b3948a7decdf7a00 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "16b947b26d5b42b5b3948a7decdf7a00" member_type: VOTER last_known_addr { host: "127.6.235.126" port: 40897 } }
I20260928 02:11:02.335328 10211 raft_consensus.cc:374] T 00000000000000000000000000000000 P 16b947b26d5b42b5b3948a7decdf7a00 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:11:02.335438 10211 raft_consensus.cc:729] T 00000000000000000000000000000000 P 16b947b26d5b42b5b3948a7decdf7a00 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 16b947b26d5b42b5b3948a7decdf7a00, State: Initialized, Role: FOLLOWER
I20260928 02:11:02.336090 10211 consensus_queue.cc:260] T 00000000000000000000000000000000 P 16b947b26d5b42b5b3948a7decdf7a00 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "16b947b26d5b42b5b3948a7decdf7a00" member_type: VOTER last_known_addr { host: "127.6.235.126" port: 40897 } }
I20260928 02:11:02.336356 10211 raft_consensus.cc:388] T 00000000000000000000000000000000 P 16b947b26d5b42b5b3948a7decdf7a00 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 02:11:02.336469 10211 raft_consensus.cc:482] T 00000000000000000000000000000000 P 16b947b26d5b42b5b3948a7decdf7a00 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 02:11:02.336592 10211 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 16b947b26d5b42b5b3948a7decdf7a00 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:11:02.339221 10211 raft_consensus.cc:504] T 00000000000000000000000000000000 P 16b947b26d5b42b5b3948a7decdf7a00 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "16b947b26d5b42b5b3948a7decdf7a00" member_type: VOTER last_known_addr { host: "127.6.235.126" port: 40897 } }
I20260928 02:11:02.339681 10211 leader_election.cc:302] T 00000000000000000000000000000000 P 16b947b26d5b42b5b3948a7decdf7a00 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 16b947b26d5b42b5b3948a7decdf7a00; no voters:
I20260928 02:11:02.340210 10211 leader_election.cc:288] T 00000000000000000000000000000000 P 16b947b26d5b42b5b3948a7decdf7a00 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 02:11:02.340379 10216 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 16b947b26d5b42b5b3948a7decdf7a00 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:11:02.341107 10216 raft_consensus.cc:686] T 00000000000000000000000000000000 P 16b947b26d5b42b5b3948a7decdf7a00 [term 1 LEADER]: Becoming Leader. State: Replica: 16b947b26d5b42b5b3948a7decdf7a00, State: Running, Role: LEADER
I20260928 02:11:02.341642 10216 consensus_queue.cc:237] T 00000000000000000000000000000000 P 16b947b26d5b42b5b3948a7decdf7a00 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "16b947b26d5b42b5b3948a7decdf7a00" member_type: VOTER last_known_addr { host: "127.6.235.126" port: 40897 } }
I20260928 02:11:02.342428 10211 sys_catalog.cc:564] T 00000000000000000000000000000000 P 16b947b26d5b42b5b3948a7decdf7a00 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 02:11:02.345130 10218 sys_catalog.cc:455] T 00000000000000000000000000000000 P 16b947b26d5b42b5b3948a7decdf7a00 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 16b947b26d5b42b5b3948a7decdf7a00. Latest consensus state: current_term: 1 leader_uuid: "16b947b26d5b42b5b3948a7decdf7a00" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "16b947b26d5b42b5b3948a7decdf7a00" member_type: VOTER last_known_addr { host: "127.6.235.126" port: 40897 } } }
I20260928 02:11:02.345523 10218 sys_catalog.cc:458] T 00000000000000000000000000000000 P 16b947b26d5b42b5b3948a7decdf7a00 [sys.catalog]: This master's current role is: LEADER
I20260928 02:11:02.346753 10225 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 02:11:02.346626 10217 sys_catalog.cc:455] T 00000000000000000000000000000000 P 16b947b26d5b42b5b3948a7decdf7a00 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "16b947b26d5b42b5b3948a7decdf7a00" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "16b947b26d5b42b5b3948a7decdf7a00" member_type: VOTER last_known_addr { host: "127.6.235.126" port: 40897 } } }
I20260928 02:11:02.347587 10217 sys_catalog.cc:458] T 00000000000000000000000000000000 P 16b947b26d5b42b5b3948a7decdf7a00 [sys.catalog]: This master's current role is: LEADER
I20260928 02:11:02.354625 10225 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 02:11:02.365945 10225 catalog_manager.cc:1347] Generated new cluster ID: 701f69a9a9014f0db5420327b5815a23
I20260928 02:11:02.366084 10225 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 02:11:02.387604 10225 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 02:11:02.389763 10225 catalog_manager.cc:1504] Loading token signing keys...
I20260928 02:11:02.398715 10225 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 16b947b26d5b42b5b3948a7decdf7a00: Generated new TSK 0
I20260928 02:11:02.399611 10225 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.6.235.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.235.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.235.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790561432251112-7085-0/krb5kdc/kudu_127.6.235.65.keytab.
Entry for principal kudu/127.6.235.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790561432251112-7085-0/krb5kdc/kudu_127.6.235.65.keytab.
I20260928 02:11:02.453801 7085 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTlaUuc/build/asan/bin/kudu
/tmp/dist-test-taskTlaUuc/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790561432251112-7085-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790561432251112-7085-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790561432251112-7085-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790561432251112-7085-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.235.65:0
--local_ip_for_outbound_sockets=127.6.235.65
--webserver_interface=127.6.235.65
--webserver_port=0
--tserver_master_addrs=127.6.235.126:40897
--keytab_file=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790561432251112-7085-0/krb5kdc/kudu_127.6.235.65.keytab
--principal=kudu/127.6.235.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.235.84:36177
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790561432251112-7085-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790561432251112-7085-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790561432251112-7085-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:11:02.802050 10237 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:11:02.802410 10237 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:11:02.802587 10237 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:11:02.809916 10237 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:11:02.810075 10237 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:11:02.810269 10237 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.235.65
I20260928 02:11:02.820176 10237 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.235.84:36177
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790561432251112-7085-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790561432251112-7085-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790561432251112-7085-0/krb5kdc/kudu_127.6.235.65.keytab
--principal=kudu/127.6.235.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.235.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790561432251112-7085-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.6.235.65
--webserver_port=0
--tserver_master_addrs=127.6.235.126:40897
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.235.65
--log_dir=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790561432251112-7085-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 00:51:59 UTC on ccycloud.quasar-axgkwn.root.comops.site
build id 4676464
ASAN enabled
I20260928 02:11:02.821585 10237 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:11:02.822943 10237 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:11:02.837148 10242 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:11:02.837190 10245 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:11:02.837224 10243 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:11:02.838184 10237 server_base.cc:1034] running on GCE node
I20260928 02:11:02.838943 10237 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:11:02.839942 10237 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:11:02.841182 10237 hybrid_clock.cc:648] HybridClock initialized: now 1790561462841125 us; error 39 us; skew 500 ppm
I20260928 02:11:02.841642 10237 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 02:11:02 dist-test-slave-86gv krb5kdc[10131](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561462, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.235.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:11:02.845571 10237 init.cc:377] Logged in from keytab as kudu/127.6.235.65@KRBTEST.COM (short username kudu)
I20260928 02:11:02.847293 10237 webserver.cc:460] Webserver started at http://127.6.235.65:42473/ using document root <none> and password file <none>
I20260928 02:11:02.848026 10237 fs_manager.cc:359] Metadata directory not provided
I20260928 02:11:02.848131 10237 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:11:02.848412 10237 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:11:02.851609 10237 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790561432251112-7085-0/minicluster-data/ts-0/data/instance:
uuid: "a20768da447e48b6942f37c8197cc5e2"
format_stamp: "Formatted at 2026-09-28 02:11:02 on dist-test-slave-86gv"
server_key: "bf48722069c4888f729c8320a10f63bd"
server_key_iv: "615fe058ab20aed8f21f599420a61217"
server_key_version: "encryptionkey@0"
I20260928 02:11:02.852777 10237 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790561432251112-7085-0/minicluster-data/ts-0/wal/instance:
uuid: "a20768da447e48b6942f37c8197cc5e2"
format_stamp: "Formatted at 2026-09-28 02:11:02 on dist-test-slave-86gv"
server_key: "bf48722069c4888f729c8320a10f63bd"
server_key_iv: "615fe058ab20aed8f21f599420a61217"
server_key_version: "encryptionkey@0"
I20260928 02:11:02.858570 10237 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260928 02:11:02.862543 10252 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:11:02.864025 10237 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260928 02:11:02.864228 10237 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790561432251112-7085-0/minicluster-data/ts-0/data,/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790561432251112-7085-0/minicluster-data/ts-0/wal
uuid: "a20768da447e48b6942f37c8197cc5e2"
format_stamp: "Formatted at 2026-09-28 02:11:02 on dist-test-slave-86gv"
server_key: "bf48722069c4888f729c8320a10f63bd"
server_key_iv: "615fe058ab20aed8f21f599420a61217"
server_key_version: "encryptionkey@0"
I20260928 02:11:02.864456 10237 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790561432251112-7085-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790561432251112-7085-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790561432251112-7085-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:11:02.878260 10237 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:11:02.882649 10237 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:11:02.882965 10237 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:11:02.883950 10237 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:11:02.885458 10237 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:11:02.885531 10237 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:11:02.885672 10237 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:11:02.885736 10237 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:11:02.919593 10237 rpc_server.cc:307] RPC server started. Bound to: 127.6.235.65:36859
I20260928 02:11:02.919631 10364 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.235.65:36859 every 8 connection(s)
I20260928 02:11:02.920876 10237 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790561432251112-7085-0/minicluster-data/ts-0/data/info.pb
I20260928 02:11:02.926131 7085 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTlaUuc/build/asan/bin/kudu as pid 10237
I20260928 02:11:02.926362 7085 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790561432251112-7085-0/minicluster-data/ts-0/wal/instance
I20260928 02:11:02.927166 7085 external_mini_cluster.cc:1428] Setting key 9562580a43eea2a558b6a90a8b254997
Sep 28 02:11:02 dist-test-slave-86gv krb5kdc[10131](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561462, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.235.65@KRBTEST.COM for kudu/127.6.235.126@KRBTEST.COM
I20260928 02:11:02.945268 10367 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:11:02.923829 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.235.65:48621 (local address 127.6.235.126:40897)
0928 02:11:02.924286 (+ 457us) server_negotiation.cc:206] Beginning negotiation
0928 02:11:02.924290 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:11:02.925404 (+ 1114us) server_negotiation.cc:407] Connection header received
0928 02:11:02.928031 (+ 2627us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:11:02.928037 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:11:02.928133 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:11:02.928264 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:11:02.930256 (+ 1992us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:11:02.931144 (+ 888us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:11:02.933709 (+ 2565us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:11:02.934102 (+ 393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:11:02.934200 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:11:02.937889 (+ 3689us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:11:02.937898 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:11:02.937906 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:11:02.937953 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:11:02.941589 (+ 3636us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:11:02.942400 (+ 811us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:11:02.942408 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:11:02.942413 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:11:02.942570 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:11:02.943169 (+ 599us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:11:02.943205 (+ 36us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:11:02.943212 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:11:02.943630 (+ 418us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:11:02.943779 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0928 02:11:02.944638 (+ 859us) server_negotiation.cc:299] Negotiation successful
0928 02:11:02.944862 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":344,"thread_start_us":256,"threads_started":1}
I20260928 02:11:02.946440 10368 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:11:02.924323 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.235.126:40897 (local address 127.6.235.65:48621)
0928 02:11:02.925252 (+ 929us) negotiation.cc:107] Waiting for socket to connect
0928 02:11:02.925291 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0928 02:11:02.927726 (+ 2435us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:11:02.928515 (+ 789us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:11:02.928523 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:11:02.929154 (+ 631us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:11:02.930017 (+ 863us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:11:02.930043 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:11:02.931310 (+ 1267us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:11:02.931318 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:11:02.933556 (+ 2238us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:11:02.933573 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:11:02.934331 (+ 758us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:11:02.934340 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:11:02.934662 (+ 322us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:11:02.935422 (+ 760us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:11:02.935456 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:11:02.937648 (+ 2192us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:11:02.941787 (+ 4139us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:11:02.941796 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:11:02.941811 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:11:02.942251 (+ 440us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:11:02.942697 (+ 446us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:11:02.942701 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:11:02.942703 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:11:02.943011 (+ 308us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:11:02.943775 (+ 764us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:11:02.943783 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:11:02.944252 (+ 469us) client_negotiation.cc:769] Sending connection context
0928 02:11:02.944528 (+ 276us) client_negotiation.cc:240] Negotiation successful
0928 02:11:02.945115 (+ 587us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":691,"thread_start_us":421,"threads_started":1}
I20260928 02:11:02.948189 10365 heartbeater.cc:347] Connected to a master server at 127.6.235.126:40897
I20260928 02:11:02.948516 10365 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.6.235.66@KRBTEST.COM; defaulting to no policy
I20260928 02:11:02.949343 10365 heartbeater.cc:511] Master 127.6.235.126:40897 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.235.66@KRBTEST.COM" created.
I20260928 02:11:02.951885 10176 ts_manager.cc:194] Registered new tserver with Master: a20768da447e48b6942f37c8197cc5e2 (127.6.235.65:36859)
I20260928 02:11:02.953459 10176 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.235.65@KRBTEST.COM'} at 127.6.235.65:48621
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.235.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790561432251112-7085-0/krb5kdc/kudu_127.6.235.66.keytab.
Entry for principal kudu/127.6.235.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790561432251112-7085-0/krb5kdc/kudu_127.6.235.66.keytab.
I20260928 02:11:02.977362 7085 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTlaUuc/build/asan/bin/kudu
/tmp/dist-test-taskTlaUuc/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790561432251112-7085-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790561432251112-7085-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790561432251112-7085-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790561432251112-7085-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.235.66:0
--local_ip_for_outbound_sockets=127.6.235.66
--webserver_interface=127.6.235.66
--webserver_port=0
--tserver_master_addrs=127.6.235.126:40897
--keytab_file=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790561432251112-7085-0/krb5kdc/kudu_127.6.235.66.keytab
--principal=kudu/127.6.235.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.235.84:36177
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790561432251112-7085-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790561432251112-7085-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790561432251112-7085-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:11:03.315126 10371 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:11:03.315416 10371 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:11:03.315541 10371 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:11:03.322973 10371 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:11:03.323101 10371 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:11:03.323215 10371 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.235.66
I20260928 02:11:03.333343 10371 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.235.84:36177
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790561432251112-7085-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790561432251112-7085-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790561432251112-7085-0/krb5kdc/kudu_127.6.235.66.keytab
--principal=kudu/127.6.235.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.235.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790561432251112-7085-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.6.235.66
--webserver_port=0
--tserver_master_addrs=127.6.235.126:40897
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.235.66
--log_dir=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790561432251112-7085-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 00:51:59 UTC on ccycloud.quasar-axgkwn.root.comops.site
build id 4676464
ASAN enabled
I20260928 02:11:03.334682 10371 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:11:03.336036 10371 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:11:03.350169 10377 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:11:03.350555 10376 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:11:03.350428 10379 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:11:03.351610 10371 server_base.cc:1034] running on GCE node
I20260928 02:11:03.352290 10371 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:11:03.353509 10371 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:11:03.354730 10371 hybrid_clock.cc:648] HybridClock initialized: now 1790561463354674 us; error 54 us; skew 500 ppm
I20260928 02:11:03.355083 10371 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 02:11:03 dist-test-slave-86gv krb5kdc[10131](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561463, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.235.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:11:03.358465 10371 init.cc:377] Logged in from keytab as kudu/127.6.235.66@KRBTEST.COM (short username kudu)
I20260928 02:11:03.360224 10371 webserver.cc:460] Webserver started at http://127.6.235.66:41353/ using document root <none> and password file <none>
I20260928 02:11:03.361114 10371 fs_manager.cc:359] Metadata directory not provided
I20260928 02:11:03.361266 10371 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:11:03.361595 10371 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:11:03.364691 10371 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790561432251112-7085-0/minicluster-data/ts-1/data/instance:
uuid: "b3500dee20c040989e4bdc5ac95503ff"
format_stamp: "Formatted at 2026-09-28 02:11:03 on dist-test-slave-86gv"
server_key: "de013436ed314d130726ec58053eb5ae"
server_key_iv: "c775698a251bce8d36e21c2ba06654bf"
server_key_version: "encryptionkey@0"
I20260928 02:11:03.365682 10371 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790561432251112-7085-0/minicluster-data/ts-1/wal/instance:
uuid: "b3500dee20c040989e4bdc5ac95503ff"
format_stamp: "Formatted at 2026-09-28 02:11:03 on dist-test-slave-86gv"
server_key: "de013436ed314d130726ec58053eb5ae"
server_key_iv: "c775698a251bce8d36e21c2ba06654bf"
server_key_version: "encryptionkey@0"
I20260928 02:11:03.371441 10371 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260928 02:11:03.376449 10386 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:11:03.378268 10371 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 02:11:03.378471 10371 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790561432251112-7085-0/minicluster-data/ts-1/data,/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790561432251112-7085-0/minicluster-data/ts-1/wal
uuid: "b3500dee20c040989e4bdc5ac95503ff"
format_stamp: "Formatted at 2026-09-28 02:11:03 on dist-test-slave-86gv"
server_key: "de013436ed314d130726ec58053eb5ae"
server_key_iv: "c775698a251bce8d36e21c2ba06654bf"
server_key_version: "encryptionkey@0"
I20260928 02:11:03.378692 10371 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790561432251112-7085-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790561432251112-7085-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790561432251112-7085-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:11:03.419034 10371 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:11:03.423775 10371 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:11:03.424216 10371 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:11:03.425244 10371 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:11:03.426642 10371 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:11:03.426723 10371 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:11:03.426868 10371 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:11:03.426929 10371 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:11:03.461506 10371 rpc_server.cc:307] RPC server started. Bound to: 127.6.235.66:41379
I20260928 02:11:03.461705 10499 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.235.66:41379 every 8 connection(s)
I20260928 02:11:03.462858 10371 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790561432251112-7085-0/minicluster-data/ts-1/data/info.pb
I20260928 02:11:03.468777 7085 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTlaUuc/build/asan/bin/kudu as pid 10371
I20260928 02:11:03.469003 7085 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790561432251112-7085-0/minicluster-data/ts-1/wal/instance
I20260928 02:11:03.469425 7085 external_mini_cluster.cc:1428] Setting key f42b1e1cc71b67392d0cc6722f149f84
Sep 28 02:11:03 dist-test-slave-86gv krb5kdc[10131](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561463, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.235.66@KRBTEST.COM for kudu/127.6.235.126@KRBTEST.COM
I20260928 02:11:03.484638 10502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:11:03.465926 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.235.66:43897 (local address 127.6.235.126:40897)
0928 02:11:03.466447 (+ 521us) server_negotiation.cc:206] Beginning negotiation
0928 02:11:03.466453 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 02:11:03.467435 (+ 982us) server_negotiation.cc:407] Connection header received
0928 02:11:03.469610 (+ 2175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:11:03.469624 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:11:03.469695 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:11:03.469818 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:11:03.471557 (+ 1739us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:11:03.472238 (+ 681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:11:03.473888 (+ 1650us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:11:03.474299 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:11:03.474444 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:11:03.478264 (+ 3820us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:11:03.478269 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:11:03.478273 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:11:03.478306 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:11:03.481150 (+ 2844us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:11:03.481950 (+ 800us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:11:03.481955 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:11:03.481957 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:11:03.482122 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:11:03.482566 (+ 444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:11:03.482571 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:11:03.482574 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:11:03.482899 (+ 325us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:11:03.483051 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0928 02:11:03.484003 (+ 952us) server_negotiation.cc:299] Negotiation successful
0928 02:11:03.484220 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":406,"thread_start_us":279,"threads_started":1}
I20260928 02:11:03.485728 10503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:11:03.466540 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.235.126:40897 (local address 127.6.235.66:43897)
0928 02:11:03.467241 (+ 701us) negotiation.cc:107] Waiting for socket to connect
0928 02:11:03.467286 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0928 02:11:03.469247 (+ 1961us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:11:03.470062 (+ 815us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:11:03.470071 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:11:03.470709 (+ 638us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:11:03.471402 (+ 693us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:11:03.471419 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:11:03.472396 (+ 977us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:11:03.472406 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:11:03.473721 (+ 1315us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:11:03.473733 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:11:03.474528 (+ 795us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:11:03.474535 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:11:03.474766 (+ 231us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:11:03.475516 (+ 750us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:11:03.475538 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:11:03.478006 (+ 2468us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:11:03.481311 (+ 3305us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:11:03.481321 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:11:03.481341 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:11:03.481801 (+ 460us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:11:03.482232 (+ 431us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:11:03.482238 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:11:03.482241 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:11:03.482441 (+ 200us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:11:03.483030 (+ 589us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:11:03.483039 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:11:03.483554 (+ 515us) client_negotiation.cc:769] Sending connection context
0928 02:11:03.483912 (+ 358us) client_negotiation.cc:240] Negotiation successful
0928 02:11:03.484514 (+ 602us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":476,"thread_start_us":290,"threads_started":1}
I20260928 02:11:03.487236 10500 heartbeater.cc:347] Connected to a master server at 127.6.235.126:40897
I20260928 02:11:03.487586 10500 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.6.235.67@KRBTEST.COM; defaulting to no policy
I20260928 02:11:03.488442 10500 heartbeater.cc:511] Master 127.6.235.126:40897 requested a full tablet report, sending...
I20260928 02:11:03.489969 10176 ts_manager.cc:194] Registered new tserver with Master: b3500dee20c040989e4bdc5ac95503ff (127.6.235.66:41379)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.235.67@KRBTEST.COM" created.
I20260928 02:11:03.490849 10176 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.235.66@KRBTEST.COM'} at 127.6.235.66:43897
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.235.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790561432251112-7085-0/krb5kdc/kudu_127.6.235.67.keytab.
Entry for principal kudu/127.6.235.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790561432251112-7085-0/krb5kdc/kudu_127.6.235.67.keytab.
I20260928 02:11:03.516672 7085 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTlaUuc/build/asan/bin/kudu
/tmp/dist-test-taskTlaUuc/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790561432251112-7085-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790561432251112-7085-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790561432251112-7085-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790561432251112-7085-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.235.67:0
--local_ip_for_outbound_sockets=127.6.235.67
--webserver_interface=127.6.235.67
--webserver_port=0
--tserver_master_addrs=127.6.235.126:40897
--keytab_file=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790561432251112-7085-0/krb5kdc/kudu_127.6.235.67.keytab
--principal=kudu/127.6.235.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.235.84:36177
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790561432251112-7085-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790561432251112-7085-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790561432251112-7085-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:11:03.851078 10506 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:11:03.851392 10506 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:11:03.851547 10506 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:11:03.859179 10506 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:11:03.859361 10506 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:11:03.859508 10506 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.235.67
I20260928 02:11:03.869709 10506 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.235.84:36177
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790561432251112-7085-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790561432251112-7085-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790561432251112-7085-0/krb5kdc/kudu_127.6.235.67.keytab
--principal=kudu/127.6.235.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.235.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790561432251112-7085-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.6.235.67
--webserver_port=0
--tserver_master_addrs=127.6.235.126:40897
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.235.67
--log_dir=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790561432251112-7085-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 00:51:59 UTC on ccycloud.quasar-axgkwn.root.comops.site
build id 4676464
ASAN enabled
I20260928 02:11:03.871029 10506 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:11:03.872359 10506 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:11:03.886953 10511 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:11:03.886859 10514 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:11:03.886917 10512 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:11:03.887943 10506 server_base.cc:1034] running on GCE node
I20260928 02:11:03.888681 10506 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:11:03.889782 10506 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:11:03.891036 10506 hybrid_clock.cc:648] HybridClock initialized: now 1790561463890959 us; error 64 us; skew 500 ppm
I20260928 02:11:03.891387 10506 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 02:11:03 dist-test-slave-86gv krb5kdc[10131](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561463, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.235.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:11:03.895041 10506 init.cc:377] Logged in from keytab as kudu/127.6.235.67@KRBTEST.COM (short username kudu)
I20260928 02:11:03.896641 10506 webserver.cc:460] Webserver started at http://127.6.235.67:46221/ using document root <none> and password file <none>
I20260928 02:11:03.897496 10506 fs_manager.cc:359] Metadata directory not provided
I20260928 02:11:03.897657 10506 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:11:03.898002 10506 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:11:03.901252 10506 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790561432251112-7085-0/minicluster-data/ts-2/data/instance:
uuid: "5d2870cf96da490aad922a936e8b44cd"
format_stamp: "Formatted at 2026-09-28 02:11:03 on dist-test-slave-86gv"
server_key: "b262fff3e29d334f0e80cff221194f3f"
server_key_iv: "043819a2967f2e2c575939d5c5b4ef14"
server_key_version: "encryptionkey@0"
I20260928 02:11:03.902124 10506 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790561432251112-7085-0/minicluster-data/ts-2/wal/instance:
uuid: "5d2870cf96da490aad922a936e8b44cd"
format_stamp: "Formatted at 2026-09-28 02:11:03 on dist-test-slave-86gv"
server_key: "b262fff3e29d334f0e80cff221194f3f"
server_key_iv: "043819a2967f2e2c575939d5c5b4ef14"
server_key_version: "encryptionkey@0"
I20260928 02:11:03.907729 10506 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260928 02:11:03.911614 10521 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:11:03.913201 10506 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260928 02:11:03.913395 10506 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790561432251112-7085-0/minicluster-data/ts-2/data,/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790561432251112-7085-0/minicluster-data/ts-2/wal
uuid: "5d2870cf96da490aad922a936e8b44cd"
format_stamp: "Formatted at 2026-09-28 02:11:03 on dist-test-slave-86gv"
server_key: "b262fff3e29d334f0e80cff221194f3f"
server_key_iv: "043819a2967f2e2c575939d5c5b4ef14"
server_key_version: "encryptionkey@0"
I20260928 02:11:03.913616 10506 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790561432251112-7085-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790561432251112-7085-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790561432251112-7085-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:11:03.941386 10506 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:11:03.946302 10506 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:11:03.946625 10506 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:11:03.947695 10506 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:11:03.949093 10506 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:11:03.949200 10506 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:11:03.949361 10506 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:11:03.949436 10506 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:11:03.956769 10365 heartbeater.cc:503] Master 127.6.235.126:40897 was elected leader, sending a full tablet report...
I20260928 02:11:03.984742 10506 rpc_server.cc:307] RPC server started. Bound to: 127.6.235.67:40499
I20260928 02:11:03.984794 10633 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.235.67:40499 every 8 connection(s)
I20260928 02:11:03.986115 10506 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790561432251112-7085-0/minicluster-data/ts-2/data/info.pb
I20260928 02:11:03.987422 7085 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTlaUuc/build/asan/bin/kudu as pid 10506
I20260928 02:11:03.987618 7085 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790561432251112-7085-0/minicluster-data/ts-2/wal/instance
I20260928 02:11:03.988183 7085 external_mini_cluster.cc:1428] Setting key 9848d5d9c8b7196524aae5d80b336515
Sep 28 02:11:03 dist-test-slave-86gv krb5kdc[10131](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561463, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.235.67@KRBTEST.COM for kudu/127.6.235.126@KRBTEST.COM
I20260928 02:11:04.007385 10636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:11:03.989042 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.235.67:41253 (local address 127.6.235.126:40897)
0928 02:11:03.989557 (+ 515us) server_negotiation.cc:206] Beginning negotiation
0928 02:11:03.989565 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 02:11:03.990763 (+ 1198us) server_negotiation.cc:407] Connection header received
0928 02:11:03.992718 (+ 1955us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:11:03.992722 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:11:03.992785 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:11:03.992873 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:11:03.994468 (+ 1595us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:11:03.995235 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:11:03.996846 (+ 1611us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:11:03.997303 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:11:03.997406 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:11:04.001059 (+ 3653us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:11:04.001064 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:11:04.001067 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:11:04.001101 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:11:04.004095 (+ 2994us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:11:04.005074 (+ 979us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:11:04.005079 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:11:04.005081 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:11:04.005214 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:11:04.005712 (+ 498us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:11:04.005716 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:11:04.005718 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:11:04.005995 (+ 277us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:11:04.006152 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0928 02:11:04.006968 (+ 816us) server_negotiation.cc:299] Negotiation successful
0928 02:11:04.007126 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":392,"thread_start_us":250,"threads_started":1}
I20260928 02:11:04.008425 10637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:11:03.989726 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.235.126:40897 (local address 127.6.235.67:41253)
0928 02:11:03.990598 (+ 872us) negotiation.cc:107] Waiting for socket to connect
0928 02:11:03.990642 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0928 02:11:03.992446 (+ 1804us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:11:03.993171 (+ 725us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:11:03.993180 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:11:03.993698 (+ 518us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:11:03.994313 (+ 615us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:11:03.994330 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:11:03.995390 (+ 1060us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:11:03.995394 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:11:03.996694 (+ 1300us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:11:03.996707 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:11:03.997508 (+ 801us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:11:03.997513 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:11:03.997753 (+ 240us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:11:03.998512 (+ 759us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:11:03.998535 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:11:04.000790 (+ 2255us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:11:04.004251 (+ 3461us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:11:04.004260 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:11:04.004275 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:11:04.004809 (+ 534us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:11:04.005327 (+ 518us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:11:04.005331 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:11:04.005334 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:11:04.005565 (+ 231us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:11:04.006120 (+ 555us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:11:04.006126 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:11:04.006505 (+ 379us) client_negotiation.cc:769] Sending connection context
0928 02:11:04.006772 (+ 267us) client_negotiation.cc:240] Negotiation successful
0928 02:11:04.007193 (+ 421us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":604,"thread_start_us":323,"threads_started":1}
I20260928 02:11:04.009946 10634 heartbeater.cc:347] Connected to a master server at 127.6.235.126:40897
I20260928 02:11:04.010288 10634 heartbeater.cc:464] Registering TS with master...
I20260928 02:11:04.011080 10634 heartbeater.cc:511] Master 127.6.235.126:40897 requested a full tablet report, sending...
I20260928 02:11:04.012552 10176 ts_manager.cc:194] Registered new tserver with Master: 5d2870cf96da490aad922a936e8b44cd (127.6.235.67:40499)
I20260928 02:11:04.013438 10176 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.235.67@KRBTEST.COM'} at 127.6.235.67:41253
I20260928 02:11:04.016340 7085 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 28 02:11:04 dist-test-slave-86gv krb5kdc[10131](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561464, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for joe-interloper@KRBTEST.COM:
Sep 28 02:11:04 dist-test-slave-86gv krb5kdc[10131](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561464, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.6.235.126@KRBTEST.COM
I20260928 02:11:04.068354 10636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:11:04.053284 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58928 (local address 127.6.235.126:40897)
0928 02:11:04.053475 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0928 02:11:04.053479 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:11:04.053956 (+ 477us) server_negotiation.cc:407] Connection header received
0928 02:11:04.054145 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:11:04.054149 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:11:04.054218 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:11:04.054316 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:11:04.055748 (+ 1432us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:11:04.056498 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:11:04.057811 (+ 1313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:11:04.058208 (+ 397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:11:04.058310 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:11:04.062518 (+ 4208us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:11:04.062524 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:11:04.062527 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:11:04.062569 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:11:04.065506 (+ 2937us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:11:04.066262 (+ 756us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:11:04.066268 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:11:04.066271 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:11:04.066403 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:11:04.066789 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:11:04.066794 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:11:04.066797 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:11:04.067141 (+ 344us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:11:04.067398 (+ 257us) server_negotiation.cc:1035] Waiting for connection context
0928 02:11:04.067740 (+ 342us) server_negotiation.cc:299] Negotiation successful
0928 02:11:04.067992 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
W20260928 02:11:04.069087 10176 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.ConnectToMaster from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:58928
I20260928 02:11:04.071326 7085 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskTlaUuc/build/asan/bin/kudu pid 10237
Sep 28 02:11:04 dist-test-slave-86gv krb5kdc[10131](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561464, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.6.235.65@KRBTEST.COM
I20260928 02:11:04.089483 10646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:11:04.072436 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52942 (local address 127.6.235.65:36859)
0928 02:11:04.073168 (+ 732us) server_negotiation.cc:206] Beginning negotiation
0928 02:11:04.073176 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 02:11:04.073200 (+ 24us) server_negotiation.cc:407] Connection header received
0928 02:11:04.073258 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:11:04.073265 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:11:04.073487 (+ 222us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:11:04.073659 (+ 172us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:11:04.074968 (+ 1309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:11:04.075987 (+ 1019us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:11:04.077579 (+ 1592us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:11:04.078416 (+ 837us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:11:04.078598 (+ 182us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:11:04.083378 (+ 4780us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:11:04.083393 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:11:04.083409 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:11:04.083460 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:11:04.086669 (+ 3209us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:11:04.087472 (+ 803us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:11:04.087480 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:11:04.087483 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:11:04.087616 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:11:04.087968 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:11:04.087971 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:11:04.087973 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:11:04.088349 (+ 376us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:11:04.088618 (+ 269us) server_negotiation.cc:1035] Waiting for connection context
0928 02:11:04.088854 (+ 236us) server_negotiation.cc:299] Negotiation successful
0928 02:11:04.089156 (+ 302us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":556,"thread_start_us":302,"threads_started":1}
W20260928 02:11:04.090782 10360 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:52942
W20260928 02:11:04.091925 7085 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskTlaUuc/build/asan/bin/kudu pid 10237: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260928 02:11:04.092137 7085 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTlaUuc/build/asan/bin/kudu with pid 10237
I20260928 02:11:04.108629 7085 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskTlaUuc/build/asan/bin/kudu pid 10371
Sep 28 02:11:04 dist-test-slave-86gv krb5kdc[10131](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561464, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.6.235.66@KRBTEST.COM
I20260928 02:11:04.124969 10647 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:11:04.109599 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33780 (local address 127.6.235.66:41379)
0928 02:11:04.110167 (+ 568us) server_negotiation.cc:206] Beginning negotiation
0928 02:11:04.110174 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 02:11:04.110202 (+ 28us) server_negotiation.cc:407] Connection header received
0928 02:11:04.110264 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:11:04.110271 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:11:04.110492 (+ 221us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:11:04.110628 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:11:04.111886 (+ 1258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:11:04.112881 (+ 995us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:11:04.114235 (+ 1354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:11:04.114774 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:11:04.114870 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:11:04.119101 (+ 4231us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:11:04.119113 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:11:04.119128 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:11:04.119172 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:11:04.121949 (+ 2777us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:11:04.122687 (+ 738us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:11:04.122695 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:11:04.122697 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:11:04.122823 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:11:04.123280 (+ 457us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:11:04.123285 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:11:04.123288 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:11:04.123639 (+ 351us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:11:04.123961 (+ 322us) server_negotiation.cc:1035] Waiting for connection context
0928 02:11:04.124264 (+ 303us) server_negotiation.cc:299] Negotiation successful
0928 02:11:04.124538 (+ 274us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":416,"thread_start_us":304,"threads_started":1}
W20260928 02:11:04.126327 10495 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:33780
W20260928 02:11:04.127557 7085 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskTlaUuc/build/asan/bin/kudu pid 10371: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260928 02:11:04.127781 7085 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTlaUuc/build/asan/bin/kudu with pid 10371
I20260928 02:11:04.144651 7085 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskTlaUuc/build/asan/bin/kudu pid 10506
Sep 28 02:11:04 dist-test-slave-86gv krb5kdc[10131](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561464, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.6.235.67@KRBTEST.COM
I20260928 02:11:04.162569 10648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:11:04.145865 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55368 (local address 127.6.235.67:40499)
0928 02:11:04.146516 (+ 651us) server_negotiation.cc:206] Beginning negotiation
0928 02:11:04.146524 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 02:11:04.146554 (+ 30us) server_negotiation.cc:407] Connection header received
0928 02:11:04.146614 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:11:04.146620 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:11:04.146950 (+ 330us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:11:04.147079 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:11:04.148567 (+ 1488us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:11:04.149666 (+ 1099us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:11:04.151032 (+ 1366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:11:04.151589 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:11:04.151688 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:11:04.156406 (+ 4718us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:11:04.156421 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:11:04.156436 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:11:04.156473 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:11:04.159610 (+ 3137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:11:04.160396 (+ 786us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:11:04.160404 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:11:04.160406 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:11:04.160525 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:11:04.160922 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:11:04.160928 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:11:04.160932 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:11:04.161287 (+ 355us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:11:04.161604 (+ 317us) server_negotiation.cc:1035] Waiting for connection context
0928 02:11:04.161939 (+ 335us) server_negotiation.cc:299] Negotiation successful
0928 02:11:04.162217 (+ 278us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":512,"thread_start_us":319,"threads_started":1}
W20260928 02:11:04.163897 10629 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:55368
W20260928 02:11:04.165071 7085 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskTlaUuc/build/asan/bin/kudu pid 10506: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260928 02:11:04.165215 7085 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTlaUuc/build/asan/bin/kudu with pid 10506
I20260928 02:11:04.181646 7085 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskTlaUuc/build/asan/bin/kudu pid 10144
I20260928 02:11:04.182930 10206 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 02:11:04.287083 7085 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTlaUuc/build/asan/bin/kudu with pid 10144
2026-09-28T02:11:04Z chronyd exiting
[ OK ] SecurityITest.TestUnauthorizedClientKerberosCredentials (3731 ms)
[ RUN ] SecurityITest.TestAuthorizedSuperuser
Loading random data
Initializing database '/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790561432251112-7085-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 02:11:04 dist-test-slave-86gv krb5kdc[10652](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 02:11:04 dist-test-slave-86gv krb5kdc[10652](info): set up 2 sockets
Sep 28 02:11:04 dist-test-slave-86gv krb5kdc[10652](info): commencing operation
krb5kdc: starting...
W20260928 02:11:05.433828 7085 mini_kdc.cc:121] Time spent starting KDC: real 1.103s user 0.003s sys 0.018s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790561432251112-7085-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790561432251112-7085-0/krb5kdc/test-user.keytab.
Sep 28 02:11:05 dist-test-slave-86gv krb5kdc[10652](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561465, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T02:11:05Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T02:11:05Z Disabled control of system clock
WARNING: no policy specified for kudu/127.6.235.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.235.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.235.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790561432251112-7085-0/krb5kdc/kudu_127.6.235.126.keytab.
Entry for principal kudu/127.6.235.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790561432251112-7085-0/krb5kdc/kudu_127.6.235.126.keytab.
I20260928 02:11:05.641129 7085 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTlaUuc/build/asan/bin/kudu
/tmp/dist-test-taskTlaUuc/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790561432251112-7085-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790561432251112-7085-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790561432251112-7085-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790561432251112-7085-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.235.126:45753
--webserver_interface=127.6.235.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790561432251112-7085-0/krb5kdc/kudu_127.6.235.126.keytab
--principal=kudu/127.6.235.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.235.84:34717
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.235.126:45753
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790561432251112-7085-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790561432251112-7085-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790561432251112-7085-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:11:05.971952 10665 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:11:05.972337 10665 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:11:05.972522 10665 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:11:05.979938 10665 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 02:11:05.980118 10665 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:11:05.980204 10665 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:11:05.980321 10665 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 02:11:05.980391 10665 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 02:11:05.990379 10665 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.235.84:34717
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790561432251112-7085-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790561432251112-7085-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.235.126:45753
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790561432251112-7085-0/krb5kdc/kudu_127.6.235.126.keytab
--principal=kudu/127.6.235.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.235.126:45753
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790561432251112-7085-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.235.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790561432251112-7085-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 00:51:59 UTC on ccycloud.quasar-axgkwn.root.comops.site
build id 4676464
ASAN enabled
I20260928 02:11:05.991835 10665 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:11:05.993295 10665 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:11:06.006685 10670 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:11:06.006724 10671 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:11:06.006655 10673 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:11:06.008033 10665 server_base.cc:1034] running on GCE node
I20260928 02:11:06.009016 10665 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:11:06.010586 10665 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:11:06.011808 10665 hybrid_clock.cc:648] HybridClock initialized: now 1790561466011766 us; error 46 us; skew 500 ppm
I20260928 02:11:06.012148 10665 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 02:11:06 dist-test-slave-86gv krb5kdc[10652](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561466, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.235.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:11:06.020474 10665 init.cc:377] Logged in from keytab as kudu/127.6.235.126@KRBTEST.COM (short username kudu)
I20260928 02:11:06.022163 10665 webserver.cc:460] Webserver started at http://127.6.235.126:37009/ using document root <none> and password file <none>
I20260928 02:11:06.022877 10665 fs_manager.cc:359] Metadata directory not provided
I20260928 02:11:06.022982 10665 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:11:06.023247 10665 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:11:06.026396 10665 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790561432251112-7085-0/minicluster-data/master-0/data/instance:
uuid: "dbfb10131c2640d8858d277900c589c2"
format_stamp: "Formatted at 2026-09-28 02:11:06 on dist-test-slave-86gv"
server_key: "deeb98e5983d2cce000d7f9315eff76e"
server_key_iv: "7f440c82fecadd343ce9246bc06cbfb6"
server_key_version: "encryptionkey@0"
I20260928 02:11:06.027168 10665 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790561432251112-7085-0/minicluster-data/master-0/wal/instance:
uuid: "dbfb10131c2640d8858d277900c589c2"
format_stamp: "Formatted at 2026-09-28 02:11:06 on dist-test-slave-86gv"
server_key: "deeb98e5983d2cce000d7f9315eff76e"
server_key_iv: "7f440c82fecadd343ce9246bc06cbfb6"
server_key_version: "encryptionkey@0"
I20260928 02:11:06.032649 10665 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260928 02:11:06.036288 10680 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:11:06.037815 10665 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 02:11:06.038033 10665 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790561432251112-7085-0/minicluster-data/master-0/data,/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790561432251112-7085-0/minicluster-data/master-0/wal
uuid: "dbfb10131c2640d8858d277900c589c2"
format_stamp: "Formatted at 2026-09-28 02:11:06 on dist-test-slave-86gv"
server_key: "deeb98e5983d2cce000d7f9315eff76e"
server_key_iv: "7f440c82fecadd343ce9246bc06cbfb6"
server_key_version: "encryptionkey@0"
I20260928 02:11:06.038240 10665 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790561432251112-7085-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790561432251112-7085-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790561432251112-7085-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:11:06.059831 10665 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:11:06.064456 10665 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:11:06.064775 10665 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:11:06.083741 10665 rpc_server.cc:307] RPC server started. Bound to: 127.6.235.126:45753
I20260928 02:11:06.083762 10731 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.235.126:45753 every 8 connection(s)
I20260928 02:11:06.085224 10665 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790561432251112-7085-0/minicluster-data/master-0/data/info.pb
I20260928 02:11:06.089227 10732 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:11:06.091380 7085 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTlaUuc/build/asan/bin/kudu as pid 10665
I20260928 02:11:06.091566 7085 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790561432251112-7085-0/minicluster-data/master-0/wal/instance
I20260928 02:11:06.092227 7085 external_mini_cluster.cc:1428] Setting key f4c1b2cfb21706e42a2755b93fc5dd44
I20260928 02:11:06.102548 10732 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dbfb10131c2640d8858d277900c589c2: Bootstrap starting.
I20260928 02:11:06.106187 10732 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P dbfb10131c2640d8858d277900c589c2: Neither blocks nor log segments found. Creating new log.
I20260928 02:11:06.107218 10732 log.cc:824] T 00000000000000000000000000000000 P dbfb10131c2640d8858d277900c589c2: Log is configured to *not* fsync() on all Append() calls
Sep 28 02:11:06 dist-test-slave-86gv krb5kdc[10652](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561465, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.235.126@KRBTEST.COM
I20260928 02:11:06.110415 10732 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dbfb10131c2640d8858d277900c589c2: No bootstrap required, opened a new log
I20260928 02:11:06.117053 10735 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:11:06.094419 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35770 (local address 127.6.235.126:45753)
0928 02:11:06.095238 (+ 819us) server_negotiation.cc:206] Beginning negotiation
0928 02:11:06.095261 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0928 02:11:06.095316 (+ 55us) server_negotiation.cc:407] Connection header received
0928 02:11:06.097740 (+ 2424us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:11:06.097772 (+ 32us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:11:06.098352 (+ 580us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:11:06.098827 (+ 475us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:11:06.100170 (+ 1343us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:11:06.102206 (+ 2036us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:11:06.103562 (+ 1356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:11:06.104661 (+ 1099us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:11:06.104916 (+ 255us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:11:06.109175 (+ 4259us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:11:06.109194 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:11:06.109210 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:11:06.109252 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:11:06.112548 (+ 3296us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:11:06.113620 (+ 1072us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:11:06.113629 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:11:06.113639 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:11:06.113815 (+ 176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:11:06.114198 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:11:06.114204 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:11:06.114207 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:11:06.114751 (+ 544us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:11:06.115074 (+ 323us) server_negotiation.cc:1035] Waiting for connection context
0928 02:11:06.115497 (+ 423us) server_negotiation.cc:299] Negotiation successful
0928 02:11:06.115921 (+ 424us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":576,"thread_start_us":336,"threads_started":1}
I20260928 02:11:06.119196 10732 raft_consensus.cc:348] T 00000000000000000000000000000000 P dbfb10131c2640d8858d277900c589c2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dbfb10131c2640d8858d277900c589c2" member_type: VOTER last_known_addr { host: "127.6.235.126" port: 45753 } }
I20260928 02:11:06.119649 10732 raft_consensus.cc:374] T 00000000000000000000000000000000 P dbfb10131c2640d8858d277900c589c2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:11:06.119765 10732 raft_consensus.cc:729] T 00000000000000000000000000000000 P dbfb10131c2640d8858d277900c589c2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dbfb10131c2640d8858d277900c589c2, State: Initialized, Role: FOLLOWER
I20260928 02:11:06.120378 10732 consensus_queue.cc:260] T 00000000000000000000000000000000 P dbfb10131c2640d8858d277900c589c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dbfb10131c2640d8858d277900c589c2" member_type: VOTER last_known_addr { host: "127.6.235.126" port: 45753 } }
I20260928 02:11:06.120579 10732 raft_consensus.cc:388] T 00000000000000000000000000000000 P dbfb10131c2640d8858d277900c589c2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 02:11:06.120672 10732 raft_consensus.cc:482] T 00000000000000000000000000000000 P dbfb10131c2640d8858d277900c589c2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 02:11:06.120798 10732 raft_consensus.cc:3037] T 00000000000000000000000000000000 P dbfb10131c2640d8858d277900c589c2 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:11:06.122928 10732 raft_consensus.cc:504] T 00000000000000000000000000000000 P dbfb10131c2640d8858d277900c589c2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dbfb10131c2640d8858d277900c589c2" member_type: VOTER last_known_addr { host: "127.6.235.126" port: 45753 } }
I20260928 02:11:06.123505 10732 leader_election.cc:302] T 00000000000000000000000000000000 P dbfb10131c2640d8858d277900c589c2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: dbfb10131c2640d8858d277900c589c2; no voters:
I20260928 02:11:06.124140 10732 leader_election.cc:288] T 00000000000000000000000000000000 P dbfb10131c2640d8858d277900c589c2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 02:11:06.124297 10737 raft_consensus.cc:2781] T 00000000000000000000000000000000 P dbfb10131c2640d8858d277900c589c2 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:11:06.125401 10737 raft_consensus.cc:686] T 00000000000000000000000000000000 P dbfb10131c2640d8858d277900c589c2 [term 1 LEADER]: Becoming Leader. State: Replica: dbfb10131c2640d8858d277900c589c2, State: Running, Role: LEADER
I20260928 02:11:06.125972 10737 consensus_queue.cc:237] T 00000000000000000000000000000000 P dbfb10131c2640d8858d277900c589c2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dbfb10131c2640d8858d277900c589c2" member_type: VOTER last_known_addr { host: "127.6.235.126" port: 45753 } }
I20260928 02:11:06.126363 10732 sys_catalog.cc:564] T 00000000000000000000000000000000 P dbfb10131c2640d8858d277900c589c2 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 02:11:06.132484 10738 sys_catalog.cc:455] T 00000000000000000000000000000000 P dbfb10131c2640d8858d277900c589c2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "dbfb10131c2640d8858d277900c589c2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dbfb10131c2640d8858d277900c589c2" member_type: VOTER last_known_addr { host: "127.6.235.126" port: 45753 } } }
I20260928 02:11:06.132541 10739 sys_catalog.cc:455] T 00000000000000000000000000000000 P dbfb10131c2640d8858d277900c589c2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader dbfb10131c2640d8858d277900c589c2. Latest consensus state: current_term: 1 leader_uuid: "dbfb10131c2640d8858d277900c589c2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dbfb10131c2640d8858d277900c589c2" member_type: VOTER last_known_addr { host: "127.6.235.126" port: 45753 } } }
I20260928 02:11:06.132886 10738 sys_catalog.cc:458] T 00000000000000000000000000000000 P dbfb10131c2640d8858d277900c589c2 [sys.catalog]: This master's current role is: LEADER
I20260928 02:11:06.132975 10739 sys_catalog.cc:458] T 00000000000000000000000000000000 P dbfb10131c2640d8858d277900c589c2 [sys.catalog]: This master's current role is: LEADER
W20260928 02:11:06.138717 10751 catalog_manager.cc:1558] T 00000000000000000000000000000000 P dbfb10131c2640d8858d277900c589c2: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 02:11:06.138823 10751 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 02:11:06.138983 10752 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 02:11:06.141106 10752 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 02:11:06.150061 10752 catalog_manager.cc:1347] Generated new cluster ID: 64404f49f4b946879f6378d29bd42aac
I20260928 02:11:06.150173 10752 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 02:11:06.173830 10752 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 02:11:06.175226 10752 catalog_manager.cc:1504] Loading token signing keys...
I20260928 02:11:06.186206 10752 catalog_manager.cc:5868] T 00000000000000000000000000000000 P dbfb10131c2640d8858d277900c589c2: Generated new TSK 0
I20260928 02:11:06.187247 10752 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.6.235.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.235.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.235.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790561432251112-7085-0/krb5kdc/kudu_127.6.235.65.keytab.
Entry for principal kudu/127.6.235.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790561432251112-7085-0/krb5kdc/kudu_127.6.235.65.keytab.
I20260928 02:11:06.261790 7085 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTlaUuc/build/asan/bin/kudu
/tmp/dist-test-taskTlaUuc/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790561432251112-7085-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790561432251112-7085-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790561432251112-7085-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790561432251112-7085-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.235.65:0
--local_ip_for_outbound_sockets=127.6.235.65
--webserver_interface=127.6.235.65
--webserver_port=0
--tserver_master_addrs=127.6.235.126:45753
--keytab_file=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790561432251112-7085-0/krb5kdc/kudu_127.6.235.65.keytab
--principal=kudu/127.6.235.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.235.84:34717
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790561432251112-7085-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790561432251112-7085-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790561432251112-7085-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:11:06.604718 10758 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:11:06.605069 10758 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:11:06.605195 10758 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:11:06.612504 10758 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:11:06.612635 10758 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:11:06.612818 10758 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.235.65
I20260928 02:11:06.622915 10758 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.235.84:34717
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790561432251112-7085-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790561432251112-7085-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790561432251112-7085-0/krb5kdc/kudu_127.6.235.65.keytab
--principal=kudu/127.6.235.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.235.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790561432251112-7085-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.6.235.65
--webserver_port=0
--tserver_master_addrs=127.6.235.126:45753
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.235.65
--log_dir=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790561432251112-7085-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 00:51:59 UTC on ccycloud.quasar-axgkwn.root.comops.site
build id 4676464
ASAN enabled
I20260928 02:11:06.624243 10758 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:11:06.625612 10758 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:11:06.639593 10764 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:11:06.639564 10763 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:11:06.639618 10766 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:11:06.640444 10758 server_base.cc:1034] running on GCE node
I20260928 02:11:06.641144 10758 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:11:06.642218 10758 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:11:06.643467 10758 hybrid_clock.cc:648] HybridClock initialized: now 1790561466643380 us; error 68 us; skew 500 ppm
I20260928 02:11:06.643805 10758 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 02:11:06 dist-test-slave-86gv krb5kdc[10652](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561466, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.235.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:11:06.648406 10758 init.cc:377] Logged in from keytab as kudu/127.6.235.65@KRBTEST.COM (short username kudu)
I20260928 02:11:06.652130 10758 webserver.cc:460] Webserver started at http://127.6.235.65:45527/ using document root <none> and password file <none>
I20260928 02:11:06.653203 10758 fs_manager.cc:359] Metadata directory not provided
I20260928 02:11:06.653407 10758 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:11:06.653795 10758 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:11:06.658133 10758 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790561432251112-7085-0/minicluster-data/ts-0/data/instance:
uuid: "c30387b38fa04ce78985b54d8aab30ff"
format_stamp: "Formatted at 2026-09-28 02:11:06 on dist-test-slave-86gv"
server_key: "dcb5b3c447af01a1ec5672a777441c79"
server_key_iv: "efddda81ecaf6107f889bdba79deaab2"
server_key_version: "encryptionkey@0"
I20260928 02:11:06.659026 10758 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790561432251112-7085-0/minicluster-data/ts-0/wal/instance:
uuid: "c30387b38fa04ce78985b54d8aab30ff"
format_stamp: "Formatted at 2026-09-28 02:11:06 on dist-test-slave-86gv"
server_key: "dcb5b3c447af01a1ec5672a777441c79"
server_key_iv: "efddda81ecaf6107f889bdba79deaab2"
server_key_version: "encryptionkey@0"
I20260928 02:11:06.664729 10758 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260928 02:11:06.668576 10773 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:11:06.670071 10758 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 02:11:06.670274 10758 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790561432251112-7085-0/minicluster-data/ts-0/data,/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790561432251112-7085-0/minicluster-data/ts-0/wal
uuid: "c30387b38fa04ce78985b54d8aab30ff"
format_stamp: "Formatted at 2026-09-28 02:11:06 on dist-test-slave-86gv"
server_key: "dcb5b3c447af01a1ec5672a777441c79"
server_key_iv: "efddda81ecaf6107f889bdba79deaab2"
server_key_version: "encryptionkey@0"
I20260928 02:11:06.670513 10758 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790561432251112-7085-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790561432251112-7085-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790561432251112-7085-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:11:06.710206 10758 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:11:06.714318 10758 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:11:06.714633 10758 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:11:06.715639 10758 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:11:06.717015 10758 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:11:06.717137 10758 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:11:06.717299 10758 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:11:06.717381 10758 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:11:06.751135 10758 rpc_server.cc:307] RPC server started. Bound to: 127.6.235.65:40555
I20260928 02:11:06.751191 10885 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.235.65:40555 every 8 connection(s)
I20260928 02:11:06.752482 10758 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790561432251112-7085-0/minicluster-data/ts-0/data/info.pb
I20260928 02:11:06.755928 7085 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTlaUuc/build/asan/bin/kudu as pid 10758
I20260928 02:11:06.756151 7085 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790561432251112-7085-0/minicluster-data/ts-0/wal/instance
I20260928 02:11:06.756824 7085 external_mini_cluster.cc:1428] Setting key f69f99ee6d852b8bc67c588d5d6e3653
Sep 28 02:11:06 dist-test-slave-86gv krb5kdc[10652](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561466, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.235.65@KRBTEST.COM for kudu/127.6.235.126@KRBTEST.COM
I20260928 02:11:06.773516 10888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:11:06.755132 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.235.65:58353 (local address 127.6.235.126:45753)
0928 02:11:06.755570 (+ 438us) server_negotiation.cc:206] Beginning negotiation
0928 02:11:06.755576 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 02:11:06.756778 (+ 1202us) server_negotiation.cc:407] Connection header received
0928 02:11:06.758920 (+ 2142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:11:06.758925 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:11:06.759006 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:11:06.759124 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:11:06.760704 (+ 1580us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:11:06.761480 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:11:06.763054 (+ 1574us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:11:06.763458 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:11:06.763593 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:11:06.767661 (+ 4068us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:11:06.767666 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:11:06.767670 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:11:06.767701 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:11:06.770236 (+ 2535us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:11:06.771211 (+ 975us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:11:06.771214 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:11:06.771216 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:11:06.771324 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:11:06.771751 (+ 427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:11:06.771758 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:11:06.771762 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:11:06.772022 (+ 260us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:11:06.772172 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0928 02:11:06.772938 (+ 766us) server_negotiation.cc:299] Negotiation successful
0928 02:11:06.773160 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":322,"thread_start_us":219,"threads_started":1}
I20260928 02:11:06.774649 10889 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:11:06.755795 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.235.126:45753 (local address 127.6.235.65:58353)
0928 02:11:06.756607 (+ 812us) negotiation.cc:107] Waiting for socket to connect
0928 02:11:06.756661 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0928 02:11:06.758615 (+ 1954us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:11:06.759370 (+ 755us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:11:06.759379 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:11:06.759943 (+ 564us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:11:06.760531 (+ 588us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:11:06.760550 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:11:06.761639 (+ 1089us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:11:06.761643 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:11:06.762886 (+ 1243us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:11:06.762898 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:11:06.763684 (+ 786us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:11:06.763692 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:11:06.763927 (+ 235us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:11:06.764618 (+ 691us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:11:06.764640 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:11:06.767425 (+ 2785us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:11:06.770389 (+ 2964us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:11:06.770402 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:11:06.770424 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:11:06.771092 (+ 668us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:11:06.771442 (+ 350us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:11:06.771448 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:11:06.771451 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:11:06.771640 (+ 189us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:11:06.772148 (+ 508us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:11:06.772156 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:11:06.772514 (+ 358us) client_negotiation.cc:769] Sending connection context
0928 02:11:06.772793 (+ 279us) client_negotiation.cc:240] Negotiation successful
0928 02:11:06.773384 (+ 591us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":541,"thread_start_us":336,"threads_started":1}
WARNING: no policy specified for kudu/127.6.235.66@KRBTEST.COM; defaulting to no policy
I20260928 02:11:06.776463 10886 heartbeater.cc:347] Connected to a master server at 127.6.235.126:45753
I20260928 02:11:06.776769 10886 heartbeater.cc:464] Registering TS with master...
I20260928 02:11:06.777541 10886 heartbeater.cc:511] Master 127.6.235.126:45753 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.235.66@KRBTEST.COM" created.
I20260928 02:11:06.779960 10697 ts_manager.cc:194] Registered new tserver with Master: c30387b38fa04ce78985b54d8aab30ff (127.6.235.65:40555)
I20260928 02:11:06.782130 10697 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.235.65@KRBTEST.COM'} at 127.6.235.65:58353
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.235.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790561432251112-7085-0/krb5kdc/kudu_127.6.235.66.keytab.
Entry for principal kudu/127.6.235.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790561432251112-7085-0/krb5kdc/kudu_127.6.235.66.keytab.
I20260928 02:11:06.804730 7085 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTlaUuc/build/asan/bin/kudu
/tmp/dist-test-taskTlaUuc/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790561432251112-7085-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790561432251112-7085-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790561432251112-7085-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790561432251112-7085-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.235.66:0
--local_ip_for_outbound_sockets=127.6.235.66
--webserver_interface=127.6.235.66
--webserver_port=0
--tserver_master_addrs=127.6.235.126:45753
--keytab_file=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790561432251112-7085-0/krb5kdc/kudu_127.6.235.66.keytab
--principal=kudu/127.6.235.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.235.84:34717
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790561432251112-7085-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790561432251112-7085-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790561432251112-7085-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:11:07.142970 10892 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:11:07.143328 10892 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:11:07.143479 10892 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:11:07.151134 10892 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:11:07.151255 10892 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:11:07.151464 10892 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.235.66
I20260928 02:11:07.162204 10892 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.235.84:34717
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790561432251112-7085-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790561432251112-7085-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790561432251112-7085-0/krb5kdc/kudu_127.6.235.66.keytab
--principal=kudu/127.6.235.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.235.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790561432251112-7085-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.6.235.66
--webserver_port=0
--tserver_master_addrs=127.6.235.126:45753
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.235.66
--log_dir=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790561432251112-7085-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 00:51:59 UTC on ccycloud.quasar-axgkwn.root.comops.site
build id 4676464
ASAN enabled
I20260928 02:11:07.163638 10892 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:11:07.165026 10892 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:11:07.179888 10897 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:11:07.179947 10900 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:11:07.179919 10898 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:11:07.180689 10892 server_base.cc:1034] running on GCE node
I20260928 02:11:07.181417 10892 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:11:07.182549 10892 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:11:07.183787 10892 hybrid_clock.cc:648] HybridClock initialized: now 1790561467183740 us; error 45 us; skew 500 ppm
I20260928 02:11:07.184132 10892 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 02:11:07 dist-test-slave-86gv krb5kdc[10652](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561467, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.235.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:11:07.187674 10892 init.cc:377] Logged in from keytab as kudu/127.6.235.66@KRBTEST.COM (short username kudu)
I20260928 02:11:07.189397 10892 webserver.cc:460] Webserver started at http://127.6.235.66:33469/ using document root <none> and password file <none>
I20260928 02:11:07.190135 10892 fs_manager.cc:359] Metadata directory not provided
I20260928 02:11:07.190267 10892 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:11:07.190593 10892 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:11:07.193614 10892 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790561432251112-7085-0/minicluster-data/ts-1/data/instance:
uuid: "256124ad7a174de09e3125bd4f24ce35"
format_stamp: "Formatted at 2026-09-28 02:11:07 on dist-test-slave-86gv"
server_key: "738f7109d317811b24e3e4cf06219371"
server_key_iv: "e66f0e2e0f0df1fcdd266a9f5b7937ee"
server_key_version: "encryptionkey@0"
I20260928 02:11:07.194483 10892 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790561432251112-7085-0/minicluster-data/ts-1/wal/instance:
uuid: "256124ad7a174de09e3125bd4f24ce35"
format_stamp: "Formatted at 2026-09-28 02:11:07 on dist-test-slave-86gv"
server_key: "738f7109d317811b24e3e4cf06219371"
server_key_iv: "e66f0e2e0f0df1fcdd266a9f5b7937ee"
server_key_version: "encryptionkey@0"
I20260928 02:11:07.199733 10892 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260928 02:11:07.203231 10907 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:11:07.204643 10892 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260928 02:11:07.204806 10892 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790561432251112-7085-0/minicluster-data/ts-1/data,/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790561432251112-7085-0/minicluster-data/ts-1/wal
uuid: "256124ad7a174de09e3125bd4f24ce35"
format_stamp: "Formatted at 2026-09-28 02:11:07 on dist-test-slave-86gv"
server_key: "738f7109d317811b24e3e4cf06219371"
server_key_iv: "e66f0e2e0f0df1fcdd266a9f5b7937ee"
server_key_version: "encryptionkey@0"
I20260928 02:11:07.205088 10892 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790561432251112-7085-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790561432251112-7085-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790561432251112-7085-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:11:07.220434 10892 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:11:07.225222 10892 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:11:07.225522 10892 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:11:07.226691 10892 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:11:07.228128 10892 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:11:07.228202 10892 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:11:07.228329 10892 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:11:07.228389 10892 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:11:07.262168 10892 rpc_server.cc:307] RPC server started. Bound to: 127.6.235.66:36095
I20260928 02:11:07.262218 11019 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.235.66:36095 every 8 connection(s)
I20260928 02:11:07.263520 10892 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790561432251112-7085-0/minicluster-data/ts-1/data/info.pb
I20260928 02:11:07.265324 7085 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTlaUuc/build/asan/bin/kudu as pid 10892
I20260928 02:11:07.265501 7085 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790561432251112-7085-0/minicluster-data/ts-1/wal/instance
I20260928 02:11:07.265987 7085 external_mini_cluster.cc:1428] Setting key 59a55b23f93dab310ec9cee52c0bb95b
Sep 28 02:11:07 dist-test-slave-86gv krb5kdc[10652](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561467, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.235.66@KRBTEST.COM for kudu/127.6.235.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.6.235.67@KRBTEST.COM; defaulting to no policy
I20260928 02:11:07.286707 10888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:11:07.266443 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.235.66:55561 (local address 127.6.235.126:45753)
0928 02:11:07.266659 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0928 02:11:07.266666 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 02:11:07.268157 (+ 1491us) server_negotiation.cc:407] Connection header received
0928 02:11:07.271226 (+ 3069us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:11:07.271232 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:11:07.271307 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:11:07.271399 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:11:07.273784 (+ 2385us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:11:07.274609 (+ 825us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:11:07.276542 (+ 1933us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:11:07.277028 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:11:07.277182 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:11:07.280734 (+ 3552us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:11:07.280739 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:11:07.280743 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:11:07.280777 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:11:07.283489 (+ 2712us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:11:07.284277 (+ 788us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:11:07.284282 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:11:07.284288 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:11:07.284412 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:11:07.284843 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:11:07.284851 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:11:07.284854 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:11:07.285190 (+ 336us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:11:07.285344 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0928 02:11:07.286128 (+ 784us) server_negotiation.cc:299] Negotiation successful
0928 02:11:07.286365 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260928 02:11:07.287813 11022 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:11:07.267085 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.235.126:45753 (local address 127.6.235.66:55561)
0928 02:11:07.268007 (+ 922us) negotiation.cc:107] Waiting for socket to connect
0928 02:11:07.268054 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0928 02:11:07.270872 (+ 2818us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:11:07.271712 (+ 840us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:11:07.271727 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:11:07.272587 (+ 860us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:11:07.273562 (+ 975us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:11:07.273592 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:11:07.274834 (+ 1242us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:11:07.274837 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:11:07.276387 (+ 1550us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:11:07.276399 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:11:07.277279 (+ 880us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:11:07.277286 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:11:07.277530 (+ 244us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:11:07.278256 (+ 726us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:11:07.278280 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:11:07.280473 (+ 2193us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:11:07.283665 (+ 3192us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:11:07.283675 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:11:07.283695 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:11:07.284141 (+ 446us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:11:07.284525 (+ 384us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:11:07.284529 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:11:07.284533 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:11:07.284733 (+ 200us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:11:07.285321 (+ 588us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:11:07.285330 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:11:07.285704 (+ 374us) client_negotiation.cc:769] Sending connection context
0928 02:11:07.286038 (+ 334us) client_negotiation.cc:240] Negotiation successful
0928 02:11:07.286618 (+ 580us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":573,"thread_start_us":344,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.235.67@KRBTEST.COM" created.
I20260928 02:11:07.289356 11020 heartbeater.cc:347] Connected to a master server at 127.6.235.126:45753
I20260928 02:11:07.289708 11020 heartbeater.cc:464] Registering TS with master...
I20260928 02:11:07.290655 11020 heartbeater.cc:511] Master 127.6.235.126:45753 requested a full tablet report, sending...
I20260928 02:11:07.292346 10697 ts_manager.cc:194] Registered new tserver with Master: 256124ad7a174de09e3125bd4f24ce35 (127.6.235.66:36095)
I20260928 02:11:07.293278 10697 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.235.66@KRBTEST.COM'} at 127.6.235.66:55561
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.235.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790561432251112-7085-0/krb5kdc/kudu_127.6.235.67.keytab.
Entry for principal kudu/127.6.235.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790561432251112-7085-0/krb5kdc/kudu_127.6.235.67.keytab.
I20260928 02:11:07.317382 7085 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTlaUuc/build/asan/bin/kudu
/tmp/dist-test-taskTlaUuc/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790561432251112-7085-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790561432251112-7085-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790561432251112-7085-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790561432251112-7085-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.235.67:0
--local_ip_for_outbound_sockets=127.6.235.67
--webserver_interface=127.6.235.67
--webserver_port=0
--tserver_master_addrs=127.6.235.126:45753
--keytab_file=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790561432251112-7085-0/krb5kdc/kudu_127.6.235.67.keytab
--principal=kudu/127.6.235.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.235.84:34717
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790561432251112-7085-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790561432251112-7085-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790561432251112-7085-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:11:07.656648 11025 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:11:07.657016 11025 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:11:07.657213 11025 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:11:07.664832 11025 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:11:07.665037 11025 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:11:07.665237 11025 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.235.67
I20260928 02:11:07.675499 11025 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.235.84:34717
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790561432251112-7085-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790561432251112-7085-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790561432251112-7085-0/krb5kdc/kudu_127.6.235.67.keytab
--principal=kudu/127.6.235.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.235.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790561432251112-7085-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.6.235.67
--webserver_port=0
--tserver_master_addrs=127.6.235.126:45753
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.235.67
--log_dir=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790561432251112-7085-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 00:51:59 UTC on ccycloud.quasar-axgkwn.root.comops.site
build id 4676464
ASAN enabled
I20260928 02:11:07.676997 11025 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:11:07.678407 11025 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:11:07.691998 11033 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:11:07.692091 11031 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:11:07.692066 11030 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:11:07.693280 11025 server_base.cc:1034] running on GCE node
I20260928 02:11:07.693930 11025 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:11:07.694911 11025 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:11:07.696213 11025 hybrid_clock.cc:648] HybridClock initialized: now 1790561467696096 us; error 52 us; skew 500 ppm
I20260928 02:11:07.696563 11025 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 02:11:07 dist-test-slave-86gv krb5kdc[10652](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561467, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.235.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:11:07.699941 11025 init.cc:377] Logged in from keytab as kudu/127.6.235.67@KRBTEST.COM (short username kudu)
I20260928 02:11:07.701819 11025 webserver.cc:460] Webserver started at http://127.6.235.67:36095/ using document root <none> and password file <none>
I20260928 02:11:07.702656 11025 fs_manager.cc:359] Metadata directory not provided
I20260928 02:11:07.702834 11025 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:11:07.703182 11025 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:11:07.706359 11025 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790561432251112-7085-0/minicluster-data/ts-2/data/instance:
uuid: "93b7713db0854d2eaebc8e1bbe165987"
format_stamp: "Formatted at 2026-09-28 02:11:07 on dist-test-slave-86gv"
server_key: "1db063e04de702824b7bc216d2eab037"
server_key_iv: "6021b90bbac4211f166abe70f26050c2"
server_key_version: "encryptionkey@0"
I20260928 02:11:07.707247 11025 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790561432251112-7085-0/minicluster-data/ts-2/wal/instance:
uuid: "93b7713db0854d2eaebc8e1bbe165987"
format_stamp: "Formatted at 2026-09-28 02:11:07 on dist-test-slave-86gv"
server_key: "1db063e04de702824b7bc216d2eab037"
server_key_iv: "6021b90bbac4211f166abe70f26050c2"
server_key_version: "encryptionkey@0"
I20260928 02:11:07.713078 11025 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.008s sys 0.000s
I20260928 02:11:07.716780 11040 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:11:07.718541 11025 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260928 02:11:07.718736 11025 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790561432251112-7085-0/minicluster-data/ts-2/data,/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790561432251112-7085-0/minicluster-data/ts-2/wal
uuid: "93b7713db0854d2eaebc8e1bbe165987"
format_stamp: "Formatted at 2026-09-28 02:11:07 on dist-test-slave-86gv"
server_key: "1db063e04de702824b7bc216d2eab037"
server_key_iv: "6021b90bbac4211f166abe70f26050c2"
server_key_version: "encryptionkey@0"
I20260928 02:11:07.718967 11025 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790561432251112-7085-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790561432251112-7085-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790561432251112-7085-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:11:07.758595 11025 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:11:07.762954 11025 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:11:07.763214 11025 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:11:07.764099 11025 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:11:07.765388 11025 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:11:07.765494 11025 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:11:07.765574 11025 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:11:07.765616 11025 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:11:07.785346 10886 heartbeater.cc:503] Master 127.6.235.126:45753 was elected leader, sending a full tablet report...
I20260928 02:11:07.799093 11025 rpc_server.cc:307] RPC server started. Bound to: 127.6.235.67:35627
I20260928 02:11:07.799175 11152 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.235.67:35627 every 8 connection(s)
I20260928 02:11:07.800402 11025 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790561432251112-7085-0/minicluster-data/ts-2/data/info.pb
I20260928 02:11:07.810019 7085 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTlaUuc/build/asan/bin/kudu as pid 11025
I20260928 02:11:07.810204 7085 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790561432251112-7085-0/minicluster-data/ts-2/wal/instance
I20260928 02:11:07.810738 7085 external_mini_cluster.cc:1428] Setting key 379a49ca67cd28a86151e83cf8c09a1d
Sep 28 02:11:07 dist-test-slave-86gv krb5kdc[10652](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561467, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.235.67@KRBTEST.COM for kudu/127.6.235.126@KRBTEST.COM
I20260928 02:11:07.821523 11155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:11:07.802964 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.235.67:48243 (local address 127.6.235.126:45753)
0928 02:11:07.803450 (+ 486us) server_negotiation.cc:206] Beginning negotiation
0928 02:11:07.803457 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 02:11:07.804518 (+ 1061us) server_negotiation.cc:407] Connection header received
0928 02:11:07.806352 (+ 1834us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:11:07.806357 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:11:07.806422 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:11:07.806516 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:11:07.807990 (+ 1474us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:11:07.808730 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:11:07.810998 (+ 2268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:11:07.811465 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:11:07.811574 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:11:07.815638 (+ 4064us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:11:07.815643 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:11:07.815646 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:11:07.815681 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:11:07.818491 (+ 2810us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:11:07.819280 (+ 789us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:11:07.819284 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:11:07.819287 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:11:07.819415 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:11:07.819809 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:11:07.819813 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:11:07.819815 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:11:07.820084 (+ 269us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:11:07.820262 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0928 02:11:07.821064 (+ 802us) server_negotiation.cc:299] Negotiation successful
0928 02:11:07.821241 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":367,"thread_start_us":256,"threads_started":1}
I20260928 02:11:07.822512 11156 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:11:07.803635 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.235.126:45753 (local address 127.6.235.67:48243)
0928 02:11:07.804324 (+ 689us) negotiation.cc:107] Waiting for socket to connect
0928 02:11:07.804388 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0928 02:11:07.806087 (+ 1699us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:11:07.806720 (+ 633us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:11:07.806728 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:11:07.807234 (+ 506us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:11:07.807828 (+ 594us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:11:07.807844 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:11:07.808879 (+ 1035us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:11:07.808884 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:11:07.810804 (+ 1920us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:11:07.810818 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:11:07.811689 (+ 871us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:11:07.811696 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:11:07.811981 (+ 285us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:11:07.812859 (+ 878us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:11:07.812922 (+ 63us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:11:07.815425 (+ 2503us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:11:07.818621 (+ 3196us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:11:07.818628 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:11:07.818643 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:11:07.819096 (+ 453us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:11:07.819511 (+ 415us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:11:07.819515 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:11:07.819517 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:11:07.819701 (+ 184us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:11:07.820206 (+ 505us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:11:07.820213 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:11:07.820569 (+ 356us) client_negotiation.cc:769] Sending connection context
0928 02:11:07.820829 (+ 260us) client_negotiation.cc:240] Negotiation successful
0928 02:11:07.821325 (+ 496us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":454,"thread_start_us":268,"threads_started":1}
I20260928 02:11:07.824160 11153 heartbeater.cc:347] Connected to a master server at 127.6.235.126:45753
I20260928 02:11:07.824514 11153 heartbeater.cc:464] Registering TS with master...
I20260928 02:11:07.825409 11153 heartbeater.cc:511] Master 127.6.235.126:45753 requested a full tablet report, sending...
I20260928 02:11:07.827018 10697 ts_manager.cc:194] Registered new tserver with Master: 93b7713db0854d2eaebc8e1bbe165987 (127.6.235.67:35627)
I20260928 02:11:07.827888 10697 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.235.67@KRBTEST.COM'} at 127.6.235.67:48243
I20260928 02:11:07.838711 7085 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 28 02:11:07 dist-test-slave-86gv krb5kdc[10652](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561467, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 28 02:11:07 dist-test-slave-86gv krb5kdc[10652](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561467, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.235.65@KRBTEST.COM
I20260928 02:11:07.889870 11161 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:11:07.874278 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58754 (local address 127.6.235.65:40555)
0928 02:11:07.874989 (+ 711us) server_negotiation.cc:206] Beginning negotiation
0928 02:11:07.875000 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 02:11:07.875032 (+ 32us) server_negotiation.cc:407] Connection header received
0928 02:11:07.875113 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:11:07.875120 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:11:07.875422 (+ 302us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:11:07.875620 (+ 198us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:11:07.877048 (+ 1428us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:11:07.878040 (+ 992us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:11:07.879279 (+ 1239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:11:07.879872 (+ 593us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:11:07.880019 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:11:07.884030 (+ 4011us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:11:07.884043 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:11:07.884059 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:11:07.884099 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:11:07.886969 (+ 2870us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:11:07.887706 (+ 737us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:11:07.887714 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:11:07.887716 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:11:07.887850 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:11:07.888241 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:11:07.888247 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:11:07.888250 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:11:07.888600 (+ 350us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:11:07.888967 (+ 367us) server_negotiation.cc:1035] Waiting for connection context
0928 02:11:07.889236 (+ 269us) server_negotiation.cc:299] Negotiation successful
0928 02:11:07.889499 (+ 263us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":507,"thread_start_us":316,"threads_started":1}
Sep 28 02:11:07 dist-test-slave-86gv krb5kdc[10652](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561467, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.235.126@KRBTEST.COM
I20260928 02:11:07.908314 11155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:11:07.894724 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35782 (local address 127.6.235.126:45753)
0928 02:11:07.894957 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0928 02:11:07.894963 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 02:11:07.895113 (+ 150us) server_negotiation.cc:407] Connection header received
0928 02:11:07.895247 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:11:07.895250 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:11:07.895325 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:11:07.895428 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:11:07.896635 (+ 1207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:11:07.897545 (+ 910us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:11:07.898771 (+ 1226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:11:07.899163 (+ 392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:11:07.899298 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:11:07.903166 (+ 3868us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:11:07.903171 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:11:07.903174 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:11:07.903209 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:11:07.905665 (+ 2456us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:11:07.906334 (+ 669us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:11:07.906341 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:11:07.906345 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:11:07.906481 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:11:07.906856 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:11:07.906862 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:11:07.906865 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:11:07.907184 (+ 319us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:11:07.907442 (+ 258us) server_negotiation.cc:1035] Waiting for connection context
0928 02:11:07.907723 (+ 281us) server_negotiation.cc:299] Negotiation successful
0928 02:11:07.907962 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
W20260928 02:11:07.909166 10697 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35782
I20260928 02:11:07.910415 7085 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskTlaUuc/build/asan/bin/kudu pid 10758
I20260928 02:11:07.924207 11161 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:11:07.911249 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58760 (local address 127.6.235.65:40555)
0928 02:11:07.911437 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0928 02:11:07.911441 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:11:07.913298 (+ 1857us) server_negotiation.cc:407] Connection header received
0928 02:11:07.913420 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:11:07.913424 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:11:07.913498 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:11:07.913579 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:11:07.914704 (+ 1125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:11:07.915359 (+ 655us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:11:07.916478 (+ 1119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:11:07.916883 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:11:07.917056 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:11:07.919243 (+ 2187us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:11:07.919248 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:11:07.919251 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:11:07.919282 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:11:07.921956 (+ 2674us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:11:07.922574 (+ 618us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:11:07.922579 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:11:07.922583 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:11:07.922709 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:11:07.923055 (+ 346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:11:07.923059 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:11:07.923061 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:11:07.923301 (+ 240us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:11:07.923468 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0928 02:11:07.923760 (+ 292us) server_negotiation.cc:299] Negotiation successful
0928 02:11:07.923927 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260928 02:11:07.924777 10881 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 02:11:08.007718 7085 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTlaUuc/build/asan/bin/kudu with pid 10758
I20260928 02:11:08.025087 7085 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskTlaUuc/build/asan/bin/kudu pid 10892
Sep 28 02:11:08 dist-test-slave-86gv krb5kdc[10652](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561467, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.235.66@KRBTEST.COM
I20260928 02:11:08.041396 11166 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:11:08.026050 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40568 (local address 127.6.235.66:36095)
0928 02:11:08.026577 (+ 527us) server_negotiation.cc:206] Beginning negotiation
0928 02:11:08.026584 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 02:11:08.026608 (+ 24us) server_negotiation.cc:407] Connection header received
0928 02:11:08.026668 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:11:08.026674 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:11:08.026936 (+ 262us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:11:08.027116 (+ 180us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:11:08.028280 (+ 1164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:11:08.029597 (+ 1317us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:11:08.030856 (+ 1259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:11:08.031418 (+ 562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:11:08.031517 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:11:08.035670 (+ 4153us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:11:08.035682 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:11:08.035697 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:11:08.035733 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:11:08.038817 (+ 3084us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:11:08.039461 (+ 644us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:11:08.039468 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:11:08.039470 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:11:08.039588 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:11:08.039905 (+ 317us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:11:08.039907 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:11:08.039910 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:11:08.040232 (+ 322us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:11:08.040465 (+ 233us) server_negotiation.cc:1035] Waiting for connection context
0928 02:11:08.040724 (+ 259us) server_negotiation.cc:299] Negotiation successful
0928 02:11:08.041031 (+ 307us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":399,"thread_start_us":263,"threads_started":1}
I20260928 02:11:08.042783 11015 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 02:11:08.126823 7085 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTlaUuc/build/asan/bin/kudu with pid 10892
I20260928 02:11:08.143867 7085 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskTlaUuc/build/asan/bin/kudu pid 11025
Sep 28 02:11:08 dist-test-slave-86gv krb5kdc[10652](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561467, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.235.67@KRBTEST.COM
I20260928 02:11:08.161726 11168 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:11:08.145075 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40196 (local address 127.6.235.67:35627)
0928 02:11:08.145695 (+ 620us) server_negotiation.cc:206] Beginning negotiation
0928 02:11:08.145702 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 02:11:08.145732 (+ 30us) server_negotiation.cc:407] Connection header received
0928 02:11:08.145792 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:11:08.145798 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:11:08.146007 (+ 209us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:11:08.146144 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:11:08.147692 (+ 1548us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:11:08.148987 (+ 1295us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:11:08.150358 (+ 1371us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:11:08.150971 (+ 613us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:11:08.151081 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:11:08.156061 (+ 4980us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:11:08.156077 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:11:08.156095 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:11:08.156138 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:11:08.159084 (+ 2946us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:11:08.159788 (+ 704us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:11:08.159796 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:11:08.159798 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:11:08.159936 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:11:08.160240 (+ 304us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:11:08.160243 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:11:08.160246 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:11:08.160589 (+ 343us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:11:08.160831 (+ 242us) server_negotiation.cc:1035] Waiting for connection context
0928 02:11:08.161110 (+ 279us) server_negotiation.cc:299] Negotiation successful
0928 02:11:08.161320 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":453,"thread_start_us":301,"threads_started":1}
I20260928 02:11:08.163075 11148 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 02:11:08.247223 7085 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTlaUuc/build/asan/bin/kudu with pid 11025
I20260928 02:11:08.264424 7085 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskTlaUuc/build/asan/bin/kudu pid 10665
I20260928 02:11:08.265725 10727 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 02:11:08.368602 7085 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTlaUuc/build/asan/bin/kudu with pid 10665
2026-09-28T02:11:08Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizedSuperuser (4080 ms)
[ RUN ] SecurityITest.TestDisableWebUI
Loading random data
Initializing database '/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790561432251112-7085-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 02:11:08 dist-test-slave-86gv krb5kdc[11173](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 02:11:08 dist-test-slave-86gv krb5kdc[11173](info): set up 2 sockets
Sep 28 02:11:08 dist-test-slave-86gv krb5kdc[11173](info): commencing operation
krb5kdc: starting...
W20260928 02:11:09.485801 7085 mini_kdc.cc:121] Time spent starting KDC: real 1.074s user 0.010s sys 0.009s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790561432251112-7085-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790561432251112-7085-0/krb5kdc/test-user.keytab.
Sep 28 02:11:09 dist-test-slave-86gv krb5kdc[11173](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561469, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T02:11:09Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T02:11:09Z Disabled control of system clock
WARNING: no policy specified for kudu/127.6.235.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.235.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.235.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790561432251112-7085-0/krb5kdc/kudu_127.6.235.126.keytab.
Entry for principal kudu/127.6.235.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790561432251112-7085-0/krb5kdc/kudu_127.6.235.126.keytab.
I20260928 02:11:09.704346 7085 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTlaUuc/build/asan/bin/kudu
/tmp/dist-test-taskTlaUuc/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790561432251112-7085-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790561432251112-7085-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790561432251112-7085-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790561432251112-7085-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.235.126:44719
--webserver_interface=127.6.235.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790561432251112-7085-0/krb5kdc/kudu_127.6.235.126.keytab
--principal=kudu/127.6.235.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.235.84:43705
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.235.126:44719
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790561432251112-7085-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790561432251112-7085-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790561432251112-7085-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:11:10.038451 11186 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:11:10.038836 11186 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:11:10.039007 11186 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:11:10.046428 11186 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 02:11:10.046613 11186 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:11:10.046751 11186 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:11:10.046859 11186 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 02:11:10.046945 11186 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 02:11:10.057332 11186 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.235.84:43705
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790561432251112-7085-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790561432251112-7085-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.235.126:44719
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790561432251112-7085-0/krb5kdc/kudu_127.6.235.126.keytab
--principal=kudu/127.6.235.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.235.126:44719
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790561432251112-7085-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.235.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790561432251112-7085-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 00:51:59 UTC on ccycloud.quasar-axgkwn.root.comops.site
build id 4676464
ASAN enabled
I20260928 02:11:10.058965 11186 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:11:10.060397 11186 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:11:10.072692 11194 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:11:10.072796 11191 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:11:10.072772 11192 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:11:10.073822 11186 server_base.cc:1034] running on GCE node
I20260928 02:11:10.074648 11186 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:11:10.076207 11186 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:11:10.077445 11186 hybrid_clock.cc:648] HybridClock initialized: now 1790561470077391 us; error 57 us; skew 500 ppm
I20260928 02:11:10.077819 11186 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 02:11:10 dist-test-slave-86gv krb5kdc[11173](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561470, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.235.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:11:10.081144 11186 init.cc:377] Logged in from keytab as kudu/127.6.235.126@KRBTEST.COM (short username kudu)
I20260928 02:11:10.082379 11186 fs_manager.cc:359] Metadata directory not provided
I20260928 02:11:10.082515 11186 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:11:10.082841 11186 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:11:10.085983 11186 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790561432251112-7085-0/minicluster-data/master-0/data/instance:
uuid: "5b6f10a43e254df199f2ddcd91f77790"
format_stamp: "Formatted at 2026-09-28 02:11:10 on dist-test-slave-86gv"
server_key: "fd5521d4cdb1841acd89f1377d487ae3"
server_key_iv: "712df33222caec0419afd4d843133695"
server_key_version: "encryptionkey@0"
I20260928 02:11:10.086822 11186 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790561432251112-7085-0/minicluster-data/master-0/wal/instance:
uuid: "5b6f10a43e254df199f2ddcd91f77790"
format_stamp: "Formatted at 2026-09-28 02:11:10 on dist-test-slave-86gv"
server_key: "fd5521d4cdb1841acd89f1377d487ae3"
server_key_iv: "712df33222caec0419afd4d843133695"
server_key_version: "encryptionkey@0"
I20260928 02:11:10.091944 11186 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260928 02:11:10.095768 11200 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:11:10.097303 11186 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 02:11:10.097497 11186 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790561432251112-7085-0/minicluster-data/master-0/data,/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790561432251112-7085-0/minicluster-data/master-0/wal
uuid: "5b6f10a43e254df199f2ddcd91f77790"
format_stamp: "Formatted at 2026-09-28 02:11:10 on dist-test-slave-86gv"
server_key: "fd5521d4cdb1841acd89f1377d487ae3"
server_key_iv: "712df33222caec0419afd4d843133695"
server_key_version: "encryptionkey@0"
I20260928 02:11:10.097733 11186 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790561432251112-7085-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790561432251112-7085-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790561432251112-7085-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:11:10.113128 11186 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:11:10.117745 11186 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:11:10.118023 11186 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:11:10.137928 11186 rpc_server.cc:307] RPC server started. Bound to: 127.6.235.126:44719
I20260928 02:11:10.137961 11251 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.235.126:44719 every 8 connection(s)
I20260928 02:11:10.138896 11186 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790561432251112-7085-0/minicluster-data/master-0/data/info.pb
I20260928 02:11:10.142993 11252 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:11:10.145283 7085 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTlaUuc/build/asan/bin/kudu as pid 11186
I20260928 02:11:10.145512 7085 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790561432251112-7085-0/minicluster-data/master-0/wal/instance
I20260928 02:11:10.146376 7085 external_mini_cluster.cc:1428] Setting key d77f0bfee79bae30e7a3db1d576250c9
I20260928 02:11:10.157586 11252 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5b6f10a43e254df199f2ddcd91f77790: Bootstrap starting.
I20260928 02:11:10.162240 11252 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5b6f10a43e254df199f2ddcd91f77790: Neither blocks nor log segments found. Creating new log.
Sep 28 02:11:10 dist-test-slave-86gv krb5kdc[11173](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561469, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.235.126@KRBTEST.COM
I20260928 02:11:10.163406 11252 log.cc:824] T 00000000000000000000000000000000 P 5b6f10a43e254df199f2ddcd91f77790: Log is configured to *not* fsync() on all Append() calls
I20260928 02:11:10.166970 11252 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5b6f10a43e254df199f2ddcd91f77790: No bootstrap required, opened a new log
I20260928 02:11:10.172981 11255 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:11:10.149072 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38618 (local address 127.6.235.126:44719)
0928 02:11:10.150021 (+ 949us) server_negotiation.cc:206] Beginning negotiation
0928 02:11:10.150034 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0928 02:11:10.150078 (+ 44us) server_negotiation.cc:407] Connection header received
0928 02:11:10.151750 (+ 1672us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:11:10.151774 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:11:10.152192 (+ 418us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:11:10.152526 (+ 334us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:11:10.154623 (+ 2097us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:11:10.155991 (+ 1368us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:11:10.157970 (+ 1979us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:11:10.158840 (+ 870us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:11:10.158978 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:11:10.164219 (+ 5241us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:11:10.164227 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:11:10.164243 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:11:10.164286 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:11:10.168116 (+ 3830us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:11:10.168859 (+ 743us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:11:10.168868 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:11:10.168882 (+ 14us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:11:10.169114 (+ 232us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:11:10.169545 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:11:10.169550 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:11:10.169553 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:11:10.170334 (+ 781us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:11:10.170650 (+ 316us) server_negotiation.cc:1035] Waiting for connection context
0928 02:11:10.171023 (+ 373us) server_negotiation.cc:299] Negotiation successful
0928 02:11:10.171514 (+ 491us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":717,"thread_start_us":464,"threads_started":1}
I20260928 02:11:10.177448 11252 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5b6f10a43e254df199f2ddcd91f77790 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5b6f10a43e254df199f2ddcd91f77790" member_type: VOTER last_known_addr { host: "127.6.235.126" port: 44719 } }
I20260928 02:11:10.177879 11252 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5b6f10a43e254df199f2ddcd91f77790 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:11:10.177992 11252 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5b6f10a43e254df199f2ddcd91f77790 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5b6f10a43e254df199f2ddcd91f77790, State: Initialized, Role: FOLLOWER
I20260928 02:11:10.178623 11252 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5b6f10a43e254df199f2ddcd91f77790 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5b6f10a43e254df199f2ddcd91f77790" member_type: VOTER last_known_addr { host: "127.6.235.126" port: 44719 } }
I20260928 02:11:10.178870 11252 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5b6f10a43e254df199f2ddcd91f77790 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 02:11:10.179006 11252 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5b6f10a43e254df199f2ddcd91f77790 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 02:11:10.179186 11252 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5b6f10a43e254df199f2ddcd91f77790 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:11:10.181607 11252 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5b6f10a43e254df199f2ddcd91f77790 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5b6f10a43e254df199f2ddcd91f77790" member_type: VOTER last_known_addr { host: "127.6.235.126" port: 44719 } }
I20260928 02:11:10.182251 11252 leader_election.cc:302] T 00000000000000000000000000000000 P 5b6f10a43e254df199f2ddcd91f77790 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5b6f10a43e254df199f2ddcd91f77790; no voters:
I20260928 02:11:10.182914 11252 leader_election.cc:288] T 00000000000000000000000000000000 P 5b6f10a43e254df199f2ddcd91f77790 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 02:11:10.183050 11257 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5b6f10a43e254df199f2ddcd91f77790 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:11:10.184427 11257 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5b6f10a43e254df199f2ddcd91f77790 [term 1 LEADER]: Becoming Leader. State: Replica: 5b6f10a43e254df199f2ddcd91f77790, State: Running, Role: LEADER
I20260928 02:11:10.185053 11257 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5b6f10a43e254df199f2ddcd91f77790 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5b6f10a43e254df199f2ddcd91f77790" member_type: VOTER last_known_addr { host: "127.6.235.126" port: 44719 } }
I20260928 02:11:10.185387 11252 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5b6f10a43e254df199f2ddcd91f77790 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 02:11:10.187912 11259 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5b6f10a43e254df199f2ddcd91f77790 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5b6f10a43e254df199f2ddcd91f77790. Latest consensus state: current_term: 1 leader_uuid: "5b6f10a43e254df199f2ddcd91f77790" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5b6f10a43e254df199f2ddcd91f77790" member_type: VOTER last_known_addr { host: "127.6.235.126" port: 44719 } } }
I20260928 02:11:10.188159 11258 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5b6f10a43e254df199f2ddcd91f77790 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5b6f10a43e254df199f2ddcd91f77790" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5b6f10a43e254df199f2ddcd91f77790" member_type: VOTER last_known_addr { host: "127.6.235.126" port: 44719 } } }
I20260928 02:11:10.188386 11259 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5b6f10a43e254df199f2ddcd91f77790 [sys.catalog]: This master's current role is: LEADER
I20260928 02:11:10.188485 11258 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5b6f10a43e254df199f2ddcd91f77790 [sys.catalog]: This master's current role is: LEADER
I20260928 02:11:10.189407 11265 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 02:11:10.195199 11265 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 02:11:10.205008 11265 catalog_manager.cc:1347] Generated new cluster ID: edb15d6348184cdf8208a46c8c631043
I20260928 02:11:10.205135 11265 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 02:11:10.246275 11265 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 02:11:10.248224 11265 catalog_manager.cc:1504] Loading token signing keys...
I20260928 02:11:10.258836 11265 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5b6f10a43e254df199f2ddcd91f77790: Generated new TSK 0
I20260928 02:11:10.259951 11265 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.6.235.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.235.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.235.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790561432251112-7085-0/krb5kdc/kudu_127.6.235.65.keytab.
Entry for principal kudu/127.6.235.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790561432251112-7085-0/krb5kdc/kudu_127.6.235.65.keytab.
I20260928 02:11:10.324767 7085 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTlaUuc/build/asan/bin/kudu
/tmp/dist-test-taskTlaUuc/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790561432251112-7085-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790561432251112-7085-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790561432251112-7085-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790561432251112-7085-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.235.65:0
--local_ip_for_outbound_sockets=127.6.235.65
--webserver_interface=127.6.235.65
--webserver_port=0
--tserver_master_addrs=127.6.235.126:44719
--keytab_file=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790561432251112-7085-0/krb5kdc/kudu_127.6.235.65.keytab
--principal=kudu/127.6.235.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.235.84:43705
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790561432251112-7085-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790561432251112-7085-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790561432251112-7085-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:11:10.679241 11278 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:11:10.679610 11278 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:11:10.679803 11278 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:11:10.687680 11278 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:11:10.687872 11278 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:11:10.688067 11278 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.235.65
I20260928 02:11:10.699581 11278 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.235.84:43705
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790561432251112-7085-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790561432251112-7085-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790561432251112-7085-0/krb5kdc/kudu_127.6.235.65.keytab
--principal=kudu/127.6.235.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.235.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790561432251112-7085-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.6.235.65
--webserver_port=0
--tserver_master_addrs=127.6.235.126:44719
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.235.65
--log_dir=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790561432251112-7085-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 00:51:59 UTC on ccycloud.quasar-axgkwn.root.comops.site
build id 4676464
ASAN enabled
I20260928 02:11:10.701205 11278 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:11:10.702780 11278 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:11:10.717831 11283 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:11:10.717801 11284 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:11:10.717942 11286 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:11:10.718680 11278 server_base.cc:1034] running on GCE node
I20260928 02:11:10.719316 11278 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:11:10.720359 11278 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:11:10.721586 11278 hybrid_clock.cc:648] HybridClock initialized: now 1790561470721532 us; error 53 us; skew 500 ppm
I20260928 02:11:10.721941 11278 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 02:11:10 dist-test-slave-86gv krb5kdc[11173](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561470, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.235.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:11:10.725180 11278 init.cc:377] Logged in from keytab as kudu/127.6.235.65@KRBTEST.COM (short username kudu)
I20260928 02:11:10.726573 11278 fs_manager.cc:359] Metadata directory not provided
I20260928 02:11:10.726712 11278 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:11:10.727053 11278 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:11:10.730154 11278 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790561432251112-7085-0/minicluster-data/ts-0/data/instance:
uuid: "dbac2fc64be340f2930e9c7ed30c6a22"
format_stamp: "Formatted at 2026-09-28 02:11:10 on dist-test-slave-86gv"
server_key: "fc63ae51118607ea7d6fe7fff4a6d18e"
server_key_iv: "bed621875cb25e38fa377daa6e0d5ea5"
server_key_version: "encryptionkey@0"
I20260928 02:11:10.730988 11278 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790561432251112-7085-0/minicluster-data/ts-0/wal/instance:
uuid: "dbac2fc64be340f2930e9c7ed30c6a22"
format_stamp: "Formatted at 2026-09-28 02:11:10 on dist-test-slave-86gv"
server_key: "fc63ae51118607ea7d6fe7fff4a6d18e"
server_key_iv: "bed621875cb25e38fa377daa6e0d5ea5"
server_key_version: "encryptionkey@0"
I20260928 02:11:10.736373 11278 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260928 02:11:10.740166 11292 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:11:10.741676 11278 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 02:11:10.741870 11278 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790561432251112-7085-0/minicluster-data/ts-0/data,/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790561432251112-7085-0/minicluster-data/ts-0/wal
uuid: "dbac2fc64be340f2930e9c7ed30c6a22"
format_stamp: "Formatted at 2026-09-28 02:11:10 on dist-test-slave-86gv"
server_key: "fc63ae51118607ea7d6fe7fff4a6d18e"
server_key_iv: "bed621875cb25e38fa377daa6e0d5ea5"
server_key_version: "encryptionkey@0"
I20260928 02:11:10.742102 11278 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790561432251112-7085-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790561432251112-7085-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790561432251112-7085-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:11:10.760486 11278 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:11:10.764811 11278 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:11:10.765147 11278 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:11:10.766119 11278 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:11:10.767820 11278 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:11:10.767993 11278 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:11:10.768227 11278 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:11:10.768342 11278 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:11:10.804215 11278 rpc_server.cc:307] RPC server started. Bound to: 127.6.235.65:34679
I20260928 02:11:10.804267 11405 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.235.65:34679 every 8 connection(s)
I20260928 02:11:10.805261 11278 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790561432251112-7085-0/minicluster-data/ts-0/data/info.pb
I20260928 02:11:10.808295 7085 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTlaUuc/build/asan/bin/kudu as pid 11278
I20260928 02:11:10.808494 7085 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790561432251112-7085-0/minicluster-data/ts-0/wal/instance
I20260928 02:11:10.809369 7085 external_mini_cluster.cc:1428] Setting key d649847b3bac2dc05745cdd5de8cfba4
Sep 28 02:11:10 dist-test-slave-86gv krb5kdc[11173](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561470, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.235.65@KRBTEST.COM for kudu/127.6.235.126@KRBTEST.COM
I20260928 02:11:10.828383 11408 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:11:10.807884 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.235.65:58177 (local address 127.6.235.126:44719)
0928 02:11:10.808333 (+ 449us) server_negotiation.cc:206] Beginning negotiation
0928 02:11:10.808337 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:11:10.809670 (+ 1333us) server_negotiation.cc:407] Connection header received
0928 02:11:10.811751 (+ 2081us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:11:10.811756 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:11:10.811839 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:11:10.811964 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:11:10.813626 (+ 1662us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:11:10.814371 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:11:10.816827 (+ 2456us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:11:10.817334 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:11:10.817475 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:11:10.821627 (+ 4152us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:11:10.821632 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:11:10.821636 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:11:10.821679 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:11:10.824550 (+ 2871us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:11:10.825806 (+ 1256us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:11:10.825820 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:11:10.825822 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:11:10.825949 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:11:10.826492 (+ 543us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:11:10.826496 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:11:10.826498 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:11:10.826760 (+ 262us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:11:10.826893 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0928 02:11:10.827935 (+ 1042us) server_negotiation.cc:299] Negotiation successful
0928 02:11:10.828118 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":343,"thread_start_us":242,"threads_started":1}
WARNING: no policy specified for kudu/127.6.235.66@KRBTEST.COM; defaulting to no policy
I20260928 02:11:10.829820 11409 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:11:10.808605 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.235.126:44719 (local address 127.6.235.65:58177)
0928 02:11:10.809490 (+ 885us) negotiation.cc:107] Waiting for socket to connect
0928 02:11:10.809553 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0928 02:11:10.811474 (+ 1921us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:11:10.812207 (+ 733us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:11:10.812216 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:11:10.812751 (+ 535us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:11:10.813458 (+ 707us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:11:10.813476 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:11:10.814543 (+ 1067us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:11:10.814555 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:11:10.816605 (+ 2050us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:11:10.816623 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:11:10.817592 (+ 969us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:11:10.817598 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:11:10.817866 (+ 268us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:11:10.818650 (+ 784us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:11:10.818673 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:11:10.821334 (+ 2661us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:11:10.824745 (+ 3411us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:11:10.824760 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:11:10.824785 (+ 25us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:11:10.825607 (+ 822us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:11:10.826079 (+ 472us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:11:10.826086 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:11:10.826092 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:11:10.826367 (+ 275us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:11:10.826904 (+ 537us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:11:10.826915 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:11:10.827444 (+ 529us) client_negotiation.cc:769] Sending connection context
0928 02:11:10.827789 (+ 345us) client_negotiation.cc:240] Negotiation successful
0928 02:11:10.828451 (+ 662us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":556,"thread_start_us":344,"threads_started":1}
I20260928 02:11:10.831630 11406 heartbeater.cc:347] Connected to a master server at 127.6.235.126:44719
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.235.66@KRBTEST.COM" created.
I20260928 02:11:10.831997 11406 heartbeater.cc:464] Registering TS with master...
I20260928 02:11:10.832959 11406 heartbeater.cc:511] Master 127.6.235.126:44719 requested a full tablet report, sending...
I20260928 02:11:10.835489 11217 ts_manager.cc:194] Registered new tserver with Master: dbac2fc64be340f2930e9c7ed30c6a22 (127.6.235.65:34679)
I20260928 02:11:10.837203 11217 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.235.65@KRBTEST.COM'} at 127.6.235.65:58177
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.235.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790561432251112-7085-0/krb5kdc/kudu_127.6.235.66.keytab.
Entry for principal kudu/127.6.235.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790561432251112-7085-0/krb5kdc/kudu_127.6.235.66.keytab.
I20260928 02:11:10.860776 7085 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTlaUuc/build/asan/bin/kudu
/tmp/dist-test-taskTlaUuc/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790561432251112-7085-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790561432251112-7085-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790561432251112-7085-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790561432251112-7085-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.235.66:0
--local_ip_for_outbound_sockets=127.6.235.66
--webserver_interface=127.6.235.66
--webserver_port=0
--tserver_master_addrs=127.6.235.126:44719
--keytab_file=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790561432251112-7085-0/krb5kdc/kudu_127.6.235.66.keytab
--principal=kudu/127.6.235.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.235.84:43705
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790561432251112-7085-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790561432251112-7085-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790561432251112-7085-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:11:11.208693 11412 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:11:11.209041 11412 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:11:11.209275 11412 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:11:11.217074 11412 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:11:11.217222 11412 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:11:11.217453 11412 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.235.66
I20260928 02:11:11.227841 11412 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.235.84:43705
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790561432251112-7085-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790561432251112-7085-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790561432251112-7085-0/krb5kdc/kudu_127.6.235.66.keytab
--principal=kudu/127.6.235.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.235.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790561432251112-7085-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.6.235.66
--webserver_port=0
--tserver_master_addrs=127.6.235.126:44719
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.235.66
--log_dir=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790561432251112-7085-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 00:51:59 UTC on ccycloud.quasar-axgkwn.root.comops.site
build id 4676464
ASAN enabled
I20260928 02:11:11.229465 11412 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:11:11.230937 11412 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:11:11.245653 11420 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:11:11.245754 11418 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:11:11.245922 11417 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:11:11.246518 11412 server_base.cc:1034] running on GCE node
I20260928 02:11:11.247225 11412 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:11:11.248311 11412 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:11:11.249540 11412 hybrid_clock.cc:648] HybridClock initialized: now 1790561471249478 us; error 57 us; skew 500 ppm
I20260928 02:11:11.249964 11412 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 02:11:11 dist-test-slave-86gv krb5kdc[11173](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561471, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.235.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:11:11.258837 11412 init.cc:377] Logged in from keytab as kudu/127.6.235.66@KRBTEST.COM (short username kudu)
I20260928 02:11:11.260226 11412 fs_manager.cc:359] Metadata directory not provided
I20260928 02:11:11.260373 11412 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:11:11.260818 11412 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:11:11.264002 11412 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790561432251112-7085-0/minicluster-data/ts-1/data/instance:
uuid: "850f1d63320043e5948f1c3ce6bc9bd5"
format_stamp: "Formatted at 2026-09-28 02:11:11 on dist-test-slave-86gv"
server_key: "69535d9b092d3a33b081556facfe21e1"
server_key_iv: "deb4f187bc2eeb0e97275879c30d71d1"
server_key_version: "encryptionkey@0"
I20260928 02:11:11.264873 11412 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790561432251112-7085-0/minicluster-data/ts-1/wal/instance:
uuid: "850f1d63320043e5948f1c3ce6bc9bd5"
format_stamp: "Formatted at 2026-09-28 02:11:11 on dist-test-slave-86gv"
server_key: "69535d9b092d3a33b081556facfe21e1"
server_key_iv: "deb4f187bc2eeb0e97275879c30d71d1"
server_key_version: "encryptionkey@0"
I20260928 02:11:11.270478 11412 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260928 02:11:11.274120 11426 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:11:11.275518 11412 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260928 02:11:11.275717 11412 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790561432251112-7085-0/minicluster-data/ts-1/data,/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790561432251112-7085-0/minicluster-data/ts-1/wal
uuid: "850f1d63320043e5948f1c3ce6bc9bd5"
format_stamp: "Formatted at 2026-09-28 02:11:11 on dist-test-slave-86gv"
server_key: "69535d9b092d3a33b081556facfe21e1"
server_key_iv: "deb4f187bc2eeb0e97275879c30d71d1"
server_key_version: "encryptionkey@0"
I20260928 02:11:11.275993 11412 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790561432251112-7085-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790561432251112-7085-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790561432251112-7085-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:11:11.306327 11412 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:11:11.311088 11412 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:11:11.311399 11412 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:11:11.312419 11412 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:11:11.313949 11412 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:11:11.314075 11412 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:11:11.314204 11412 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:11:11.314288 11412 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:11:11.349547 11412 rpc_server.cc:307] RPC server started. Bound to: 127.6.235.66:39725
I20260928 02:11:11.349645 11538 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.235.66:39725 every 8 connection(s)
I20260928 02:11:11.350498 11412 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790561432251112-7085-0/minicluster-data/ts-1/data/info.pb
I20260928 02:11:11.352087 7085 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTlaUuc/build/asan/bin/kudu as pid 11412
I20260928 02:11:11.352298 7085 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790561432251112-7085-0/minicluster-data/ts-1/wal/instance
I20260928 02:11:11.353022 7085 external_mini_cluster.cc:1428] Setting key 437977b1230710199aab7f4586d40bcb
Sep 28 02:11:11 dist-test-slave-86gv krb5kdc[11173](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561471, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.235.66@KRBTEST.COM for kudu/127.6.235.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.6.235.67@KRBTEST.COM; defaulting to no policy
I20260928 02:11:11.373970 11541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:11:11.353698 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.235.66:57779 (local address 127.6.235.126:44719)
0928 02:11:11.354237 (+ 539us) server_negotiation.cc:206] Beginning negotiation
0928 02:11:11.354244 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 02:11:11.355466 (+ 1222us) server_negotiation.cc:407] Connection header received
0928 02:11:11.357419 (+ 1953us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:11:11.357428 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:11:11.357527 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:11:11.357672 (+ 145us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:11:11.359325 (+ 1653us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:11:11.360550 (+ 1225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:11:11.362449 (+ 1899us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:11:11.362914 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:11:11.363031 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:11:11.367564 (+ 4533us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:11:11.367570 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:11:11.367573 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:11:11.367610 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:11:11.370360 (+ 2750us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:11:11.371232 (+ 872us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:11:11.371242 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:11:11.371244 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:11:11.371374 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:11:11.371831 (+ 457us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:11:11.371838 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:11:11.371846 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:11:11.372167 (+ 321us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:11:11.372356 (+ 189us) server_negotiation.cc:1035] Waiting for connection context
0928 02:11:11.373456 (+ 1100us) server_negotiation.cc:299] Negotiation successful
0928 02:11:11.373627 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":380,"thread_start_us":270,"threads_started":1}
I20260928 02:11:11.375167 11542 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:11:11.354405 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.235.126:44719 (local address 127.6.235.66:57779)
0928 02:11:11.355274 (+ 869us) negotiation.cc:107] Waiting for socket to connect
0928 02:11:11.355346 (+ 72us) client_negotiation.cc:174] Beginning negotiation
0928 02:11:11.357114 (+ 1768us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:11:11.357878 (+ 764us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:11:11.357887 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:11:11.358494 (+ 607us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:11:11.359106 (+ 612us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:11:11.359125 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:11:11.360750 (+ 1625us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:11:11.360754 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:11:11.362181 (+ 1427us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:11:11.362194 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:11:11.363177 (+ 983us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:11:11.363187 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:11:11.363534 (+ 347us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:11:11.364380 (+ 846us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:11:11.364420 (+ 40us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:11:11.367280 (+ 2860us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:11:11.370537 (+ 3257us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:11:11.370549 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:11:11.370571 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:11:11.371061 (+ 490us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:11:11.371497 (+ 436us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:11:11.371502 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:11:11.371506 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:11:11.371704 (+ 198us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:11:11.372350 (+ 646us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:11:11.372363 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:11:11.372950 (+ 587us) client_negotiation.cc:769] Sending connection context
0928 02:11:11.373360 (+ 410us) client_negotiation.cc:240] Negotiation successful
0928 02:11:11.373982 (+ 622us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":593,"thread_start_us":384,"threads_started":1}
I20260928 02:11:11.376756 11539 heartbeater.cc:347] Connected to a master server at 127.6.235.126:44719
I20260928 02:11:11.377327 11539 heartbeater.cc:464] Registering TS with master...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.235.67@KRBTEST.COM" created.
I20260928 02:11:11.378149 11539 heartbeater.cc:511] Master 127.6.235.126:44719 requested a full tablet report, sending...
I20260928 02:11:11.380208 11217 ts_manager.cc:194] Registered new tserver with Master: 850f1d63320043e5948f1c3ce6bc9bd5 (127.6.235.66:39725)
I20260928 02:11:11.381235 11217 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.235.66@KRBTEST.COM'} at 127.6.235.66:57779
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.235.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790561432251112-7085-0/krb5kdc/kudu_127.6.235.67.keytab.
Entry for principal kudu/127.6.235.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790561432251112-7085-0/krb5kdc/kudu_127.6.235.67.keytab.
I20260928 02:11:11.408392 7085 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTlaUuc/build/asan/bin/kudu
/tmp/dist-test-taskTlaUuc/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790561432251112-7085-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790561432251112-7085-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790561432251112-7085-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790561432251112-7085-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.235.67:0
--local_ip_for_outbound_sockets=127.6.235.67
--webserver_interface=127.6.235.67
--webserver_port=0
--tserver_master_addrs=127.6.235.126:44719
--keytab_file=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790561432251112-7085-0/krb5kdc/kudu_127.6.235.67.keytab
--principal=kudu/127.6.235.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.235.84:43705
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790561432251112-7085-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790561432251112-7085-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790561432251112-7085-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:11:11.745993 11545 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:11:11.746379 11545 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:11:11.746492 11545 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:11:11.753981 11545 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:11:11.754102 11545 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:11:11.754227 11545 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.235.67
I20260928 02:11:11.764756 11545 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.235.84:43705
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790561432251112-7085-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790561432251112-7085-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790561432251112-7085-0/krb5kdc/kudu_127.6.235.67.keytab
--principal=kudu/127.6.235.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.235.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790561432251112-7085-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.6.235.67
--webserver_port=0
--tserver_master_addrs=127.6.235.126:44719
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.235.67
--log_dir=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790561432251112-7085-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 00:51:59 UTC on ccycloud.quasar-axgkwn.root.comops.site
build id 4676464
ASAN enabled
I20260928 02:11:11.766086 11545 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:11:11.767428 11545 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:11:11.781044 11553 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:11:11.781010 11551 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:11:11.781257 11550 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:11:11.782290 11545 server_base.cc:1034] running on GCE node
I20260928 02:11:11.783036 11545 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:11:11.784102 11545 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:11:11.785378 11545 hybrid_clock.cc:648] HybridClock initialized: now 1790561471785328 us; error 41 us; skew 500 ppm
I20260928 02:11:11.785825 11545 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 02:11:11 dist-test-slave-86gv krb5kdc[11173](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561471, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.235.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:11:11.789639 11545 init.cc:377] Logged in from keytab as kudu/127.6.235.67@KRBTEST.COM (short username kudu)
I20260928 02:11:11.790949 11545 fs_manager.cc:359] Metadata directory not provided
I20260928 02:11:11.791095 11545 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:11:11.791435 11545 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:11:11.794571 11545 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790561432251112-7085-0/minicluster-data/ts-2/data/instance:
uuid: "da365a6a0b7f4dd28c64233be2d6445e"
format_stamp: "Formatted at 2026-09-28 02:11:11 on dist-test-slave-86gv"
server_key: "da55ff72a52b74d820477a2c6e601c4f"
server_key_iv: "818fa88f6f24bc8a08a13451561bebeb"
server_key_version: "encryptionkey@0"
I20260928 02:11:11.795408 11545 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790561432251112-7085-0/minicluster-data/ts-2/wal/instance:
uuid: "da365a6a0b7f4dd28c64233be2d6445e"
format_stamp: "Formatted at 2026-09-28 02:11:11 on dist-test-slave-86gv"
server_key: "da55ff72a52b74d820477a2c6e601c4f"
server_key_iv: "818fa88f6f24bc8a08a13451561bebeb"
server_key_version: "encryptionkey@0"
I20260928 02:11:11.800644 11545 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260928 02:11:11.804311 11559 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:11:11.805841 11545 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 02:11:11.806015 11545 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790561432251112-7085-0/minicluster-data/ts-2/data,/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790561432251112-7085-0/minicluster-data/ts-2/wal
uuid: "da365a6a0b7f4dd28c64233be2d6445e"
format_stamp: "Formatted at 2026-09-28 02:11:11 on dist-test-slave-86gv"
server_key: "da55ff72a52b74d820477a2c6e601c4f"
server_key_iv: "818fa88f6f24bc8a08a13451561bebeb"
server_key_version: "encryptionkey@0"
I20260928 02:11:11.806241 11545 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790561432251112-7085-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790561432251112-7085-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790561432251112-7085-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:11:11.840672 11406 heartbeater.cc:503] Master 127.6.235.126:44719 was elected leader, sending a full tablet report...
I20260928 02:11:11.847738 11545 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:11:11.852377 11545 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:11:11.852660 11545 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:11:11.853734 11545 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:11:11.855082 11545 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:11:11.855206 11545 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:11:11.855345 11545 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:11:11.855433 11545 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:11:11.889107 11545 rpc_server.cc:307] RPC server started. Bound to: 127.6.235.67:39121
I20260928 02:11:11.889156 11671 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.235.67:39121 every 8 connection(s)
I20260928 02:11:11.890221 11545 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790561432251112-7085-0/minicluster-data/ts-2/data/info.pb
I20260928 02:11:11.899858 7085 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTlaUuc/build/asan/bin/kudu as pid 11545
I20260928 02:11:11.900058 7085 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790561432251112-7085-0/minicluster-data/ts-2/wal/instance
I20260928 02:11:11.900528 7085 external_mini_cluster.cc:1428] Setting key f07fd5588f015ef20a6d5006444a3665
Sep 28 02:11:11 dist-test-slave-86gv krb5kdc[11173](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561471, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.235.67@KRBTEST.COM for kudu/127.6.235.126@KRBTEST.COM
I20260928 02:11:11.911476 11674 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:11:11.892702 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.235.67:60407 (local address 127.6.235.126:44719)
0928 02:11:11.893175 (+ 473us) server_negotiation.cc:206] Beginning negotiation
0928 02:11:11.893180 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 02:11:11.894259 (+ 1079us) server_negotiation.cc:407] Connection header received
0928 02:11:11.896078 (+ 1819us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:11:11.896083 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:11:11.896191 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:11:11.896311 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:11:11.897938 (+ 1627us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:11:11.898696 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:11:11.900436 (+ 1740us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:11:11.901067 (+ 631us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:11:11.901218 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:11:11.905514 (+ 4296us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:11:11.905518 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:11:11.905522 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:11:11.905556 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:11:11.908213 (+ 2657us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:11:11.909042 (+ 829us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:11:11.909049 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:11:11.909053 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:11:11.909204 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:11:11.909723 (+ 519us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:11:11.909727 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:11:11.909729 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:11:11.909989 (+ 260us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:11:11.910100 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0928 02:11:11.911031 (+ 931us) server_negotiation.cc:299] Negotiation successful
0928 02:11:11.911203 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":368,"thread_start_us":258,"threads_started":1}
I20260928 02:11:11.912505 11675 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:11:11.893373 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.235.126:44719 (local address 127.6.235.67:60407)
0928 02:11:11.894102 (+ 729us) negotiation.cc:107] Waiting for socket to connect
0928 02:11:11.894141 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0928 02:11:11.895788 (+ 1647us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:11:11.896554 (+ 766us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:11:11.896563 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:11:11.897167 (+ 604us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:11:11.897750 (+ 583us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:11:11.897767 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:11:11.898899 (+ 1132us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:11:11.898903 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:11:11.900278 (+ 1375us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:11:11.900290 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:11:11.901426 (+ 1136us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:11:11.901436 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:11:11.901726 (+ 290us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:11:11.902742 (+ 1016us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:11:11.902777 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:11:11.905310 (+ 2533us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:11:11.908371 (+ 3061us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:11:11.908384 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:11:11.908399 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:11:11.908841 (+ 442us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:11:11.909355 (+ 514us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:11:11.909363 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:11:11.909367 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:11:11.909618 (+ 251us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:11:11.910095 (+ 477us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:11:11.910101 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:11:11.910560 (+ 459us) client_negotiation.cc:769] Sending connection context
0928 02:11:11.910867 (+ 307us) client_negotiation.cc:240] Negotiation successful
0928 02:11:11.911265 (+ 398us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":492,"thread_start_us":267,"threads_started":1}
I20260928 02:11:11.913959 11672 heartbeater.cc:347] Connected to a master server at 127.6.235.126:44719
I20260928 02:11:11.914275 11672 heartbeater.cc:464] Registering TS with master...
I20260928 02:11:11.915103 11672 heartbeater.cc:511] Master 127.6.235.126:44719 requested a full tablet report, sending...
I20260928 02:11:11.916617 11217 ts_manager.cc:194] Registered new tserver with Master: da365a6a0b7f4dd28c64233be2d6445e (127.6.235.67:39121)
I20260928 02:11:11.917486 11217 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.235.67@KRBTEST.COM'} at 127.6.235.67:60407
I20260928 02:11:11.928521 7085 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 02:11:11.945809 11674 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:11:11.933118 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35044 (local address 127.6.235.126:44719)
0928 02:11:11.933404 (+ 286us) server_negotiation.cc:206] Beginning negotiation
0928 02:11:11.933412 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 02:11:11.933808 (+ 396us) server_negotiation.cc:407] Connection header received
0928 02:11:11.934012 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:11:11.934017 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:11:11.934082 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:11:11.934213 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:11:11.935667 (+ 1454us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:11:11.936382 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:11:11.937704 (+ 1322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:11:11.938104 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:11:11.938237 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:11:11.940560 (+ 2323us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:11:11.940565 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:11:11.940568 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:11:11.940604 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:11:11.943098 (+ 2494us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:11:11.943810 (+ 712us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:11:11.943815 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:11:11.943817 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:11:11.943933 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:11:11.944297 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:11:11.944303 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:11:11.944306 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:11:11.944626 (+ 320us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:11:11.944879 (+ 253us) server_negotiation.cc:1035] Waiting for connection context
0928 02:11:11.945225 (+ 346us) server_negotiation.cc:299] Negotiation successful
0928 02:11:11.945454 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
I20260928 02:11:11.949983 11217 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35044:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 02:11:11.956784 11217 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 02:11:12.000566 11686 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:11:11.968407 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.235.67:39121 (local address 127.0.0.1:55740)
0928 02:11:11.988464 (+ 20057us) negotiation.cc:107] Waiting for socket to connect
0928 02:11:11.988501 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0928 02:11:11.988707 (+ 206us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:11:11.990101 (+ 1394us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:11:11.990114 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:11:11.990215 (+ 101us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:11:11.991090 (+ 875us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:11:11.991109 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:11:11.993460 (+ 2351us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:11:11.993465 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:11:11.996545 (+ 3080us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:11:11.996564 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:11:11.999191 (+ 2627us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:11:11.999202 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:11:11.999492 (+ 290us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:11:11.999576 (+ 84us) client_negotiation.cc:769] Sending connection context
0928 02:11:11.999701 (+ 125us) client_negotiation.cc:240] Negotiation successful
0928 02:11:11.999963 (+ 262us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":19778,"mutex_wait_us":48,"spinlock_wait_cycles":33408,"thread_start_us":18836,"threads_started":1}
I20260928 02:11:12.003499 11685 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:11:11.968539 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.235.66:39725 (local address 127.0.0.1:33698)
0928 02:11:11.988465 (+ 19926us) negotiation.cc:107] Waiting for socket to connect
0928 02:11:11.988501 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0928 02:11:11.988712 (+ 211us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:11:11.990103 (+ 1391us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:11:11.990114 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:11:11.990223 (+ 109us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:11:11.991016 (+ 793us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:11:11.991037 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:11:11.994969 (+ 3932us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:11:11.994977 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:11:11.999329 (+ 4352us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:11:11.999347 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:11:12.002598 (+ 3251us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:11:12.002606 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:11:12.002819 (+ 213us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:11:12.002852 (+ 33us) client_negotiation.cc:769] Sending connection context
0928 02:11:12.002921 (+ 69us) client_negotiation.cc:240] Negotiation successful
0928 02:11:12.003067 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":19726,"mutex_wait_us":34,"spinlock_wait_cycles":111488,"thread_start_us":18642,"threads_started":1}
I20260928 02:11:12.003690 11687 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:11:11.968709 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55740 (local address 127.6.235.67:39121)
0928 02:11:11.983182 (+ 14473us) server_negotiation.cc:206] Beginning negotiation
0928 02:11:11.983194 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0928 02:11:11.988952 (+ 5758us) server_negotiation.cc:407] Connection header received
0928 02:11:11.989109 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:11:11.989120 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:11:11.989414 (+ 294us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:11:11.989681 (+ 267us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:11:11.991377 (+ 1696us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:11:11.993286 (+ 1909us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:11:11.996786 (+ 3500us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:11:11.998913 (+ 2127us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:11:12.002564 (+ 3651us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:11:12.002924 (+ 360us) server_negotiation.cc:1035] Waiting for connection context
0928 02:11:12.003012 (+ 88us) server_negotiation.cc:299] Negotiation successful
0928 02:11:12.003250 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14191,"thread_start_us":14025,"threads_started":1}
I20260928 02:11:12.004613 11688 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:11:11.969040 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33698 (local address 127.6.235.66:39725)
0928 02:11:11.982543 (+ 13503us) server_negotiation.cc:206] Beginning negotiation
0928 02:11:11.982558 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0928 02:11:11.988952 (+ 6394us) server_negotiation.cc:407] Connection header received
0928 02:11:11.989083 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:11:11.989095 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:11:11.989432 (+ 337us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:11:11.989641 (+ 209us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:11:11.992850 (+ 3209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:11:11.994779 (+ 1929us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:11:11.999582 (+ 4803us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:11:12.002317 (+ 2735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:11:12.004039 (+ 1722us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:11:12.004240 (+ 201us) server_negotiation.cc:1035] Waiting for connection context
0928 02:11:12.004294 (+ 54us) server_negotiation.cc:299] Negotiation successful
0928 02:11:12.004424 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13267,"thread_start_us":14175,"threads_started":1}
I20260928 02:11:12.007725 11474 tablet_service.cc:1467] Processing CreateTablet for tablet 339b5e8b82394f30a7fa447abd6153fb (DEFAULT_TABLE table=test-table [id=963beae81a17432fbf6f6a1fbf70e924]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:11:12.008248 11607 tablet_service.cc:1467] Processing CreateTablet for tablet 339b5e8b82394f30a7fa447abd6153fb (DEFAULT_TABLE table=test-table [id=963beae81a17432fbf6f6a1fbf70e924]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:11:12.009095 11474 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 339b5e8b82394f30a7fa447abd6153fb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:11:12.009768 11607 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 339b5e8b82394f30a7fa447abd6153fb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:11:12.016716 11686 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:11:11.991649 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.235.65:34679 (local address 127.0.0.1:42390)
0928 02:11:12.001483 (+ 9834us) negotiation.cc:107] Waiting for socket to connect
0928 02:11:12.001504 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0928 02:11:12.001628 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:11:12.009603 (+ 7975us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:11:12.009608 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:11:12.009638 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:11:12.009828 (+ 190us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:11:12.009840 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:11:12.011760 (+ 1920us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:11:12.011765 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:11:12.013676 (+ 1911us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:11:12.013692 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:11:12.015868 (+ 2176us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:11:12.015874 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:11:12.016088 (+ 214us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:11:12.016139 (+ 51us) client_negotiation.cc:769] Sending connection context
0928 02:11:12.016228 (+ 89us) client_negotiation.cc:240] Negotiation successful
0928 02:11:12.016395 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9670,"thread_start_us":11124,"threads_started":1}
I20260928 02:11:12.017590 11689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:11:11.988535 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42390 (local address 127.6.235.65:34679)
0928 02:11:12.001529 (+ 12994us) server_negotiation.cc:206] Beginning negotiation
0928 02:11:12.001541 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0928 02:11:12.001583 (+ 42us) server_negotiation.cc:407] Connection header received
0928 02:11:12.001947 (+ 364us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:11:12.001960 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:11:12.009109 (+ 7149us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:11:12.009345 (+ 236us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:11:12.010016 (+ 671us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:11:12.011601 (+ 1585us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:11:12.013966 (+ 2365us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:11:12.015401 (+ 1435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:11:12.015537 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:11:12.015772 (+ 235us) server_negotiation.cc:1035] Waiting for connection context
0928 02:11:12.017098 (+ 1326us) server_negotiation.cc:299] Negotiation successful
0928 02:11:12.017273 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12776,"thread_start_us":12602,"threads_started":1}
I20260928 02:11:12.021806 11341 tablet_service.cc:1467] Processing CreateTablet for tablet 339b5e8b82394f30a7fa447abd6153fb (DEFAULT_TABLE table=test-table [id=963beae81a17432fbf6f6a1fbf70e924]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:11:12.023470 11341 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 339b5e8b82394f30a7fa447abd6153fb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:11:12.035665 11691 tablet_bootstrap.cc:492] T 339b5e8b82394f30a7fa447abd6153fb P 850f1d63320043e5948f1c3ce6bc9bd5: Bootstrap starting.
I20260928 02:11:12.041685 11692 tablet_bootstrap.cc:492] T 339b5e8b82394f30a7fa447abd6153fb P da365a6a0b7f4dd28c64233be2d6445e: Bootstrap starting.
I20260928 02:11:12.042089 11691 tablet_bootstrap.cc:654] T 339b5e8b82394f30a7fa447abd6153fb P 850f1d63320043e5948f1c3ce6bc9bd5: Neither blocks nor log segments found. Creating new log.
I20260928 02:11:12.044788 11692 tablet_bootstrap.cc:654] T 339b5e8b82394f30a7fa447abd6153fb P da365a6a0b7f4dd28c64233be2d6445e: Neither blocks nor log segments found. Creating new log.
I20260928 02:11:12.045830 11692 log.cc:824] T 339b5e8b82394f30a7fa447abd6153fb P da365a6a0b7f4dd28c64233be2d6445e: Log is configured to *not* fsync() on all Append() calls
I20260928 02:11:12.045780 11691 log.cc:824] T 339b5e8b82394f30a7fa447abd6153fb P 850f1d63320043e5948f1c3ce6bc9bd5: Log is configured to *not* fsync() on all Append() calls
I20260928 02:11:12.050192 11691 tablet_bootstrap.cc:492] T 339b5e8b82394f30a7fa447abd6153fb P 850f1d63320043e5948f1c3ce6bc9bd5: No bootstrap required, opened a new log
I20260928 02:11:12.050410 11693 tablet_bootstrap.cc:492] T 339b5e8b82394f30a7fa447abd6153fb P dbac2fc64be340f2930e9c7ed30c6a22: Bootstrap starting.
I20260928 02:11:12.050626 11691 ts_tablet_manager.cc:1397] T 339b5e8b82394f30a7fa447abd6153fb P 850f1d63320043e5948f1c3ce6bc9bd5: Time spent bootstrapping tablet: real 0.015s user 0.010s sys 0.000s
I20260928 02:11:12.055832 11693 tablet_bootstrap.cc:654] T 339b5e8b82394f30a7fa447abd6153fb P dbac2fc64be340f2930e9c7ed30c6a22: Neither blocks nor log segments found. Creating new log.
I20260928 02:11:12.055955 11692 tablet_bootstrap.cc:492] T 339b5e8b82394f30a7fa447abd6153fb P da365a6a0b7f4dd28c64233be2d6445e: No bootstrap required, opened a new log
I20260928 02:11:12.056289 11692 ts_tablet_manager.cc:1397] T 339b5e8b82394f30a7fa447abd6153fb P da365a6a0b7f4dd28c64233be2d6445e: Time spent bootstrapping tablet: real 0.015s user 0.009s sys 0.000s
I20260928 02:11:12.057615 11693 log.cc:824] T 339b5e8b82394f30a7fa447abd6153fb P dbac2fc64be340f2930e9c7ed30c6a22: Log is configured to *not* fsync() on all Append() calls
I20260928 02:11:12.062911 11691 raft_consensus.cc:348] T 339b5e8b82394f30a7fa447abd6153fb P 850f1d63320043e5948f1c3ce6bc9bd5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "da365a6a0b7f4dd28c64233be2d6445e" member_type: VOTER last_known_addr { host: "127.6.235.67" port: 39121 } } peers { permanent_uuid: "850f1d63320043e5948f1c3ce6bc9bd5" member_type: VOTER last_known_addr { host: "127.6.235.66" port: 39725 } } peers { permanent_uuid: "dbac2fc64be340f2930e9c7ed30c6a22" member_type: VOTER last_known_addr { host: "127.6.235.65" port: 34679 } }
I20260928 02:11:12.063643 11691 raft_consensus.cc:374] T 339b5e8b82394f30a7fa447abd6153fb P 850f1d63320043e5948f1c3ce6bc9bd5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:11:12.063793 11691 raft_consensus.cc:729] T 339b5e8b82394f30a7fa447abd6153fb P 850f1d63320043e5948f1c3ce6bc9bd5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 850f1d63320043e5948f1c3ce6bc9bd5, State: Initialized, Role: FOLLOWER
I20260928 02:11:12.064663 11691 consensus_queue.cc:260] T 339b5e8b82394f30a7fa447abd6153fb P 850f1d63320043e5948f1c3ce6bc9bd5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "da365a6a0b7f4dd28c64233be2d6445e" member_type: VOTER last_known_addr { host: "127.6.235.67" port: 39121 } } peers { permanent_uuid: "850f1d63320043e5948f1c3ce6bc9bd5" member_type: VOTER last_known_addr { host: "127.6.235.66" port: 39725 } } peers { permanent_uuid: "dbac2fc64be340f2930e9c7ed30c6a22" member_type: VOTER last_known_addr { host: "127.6.235.65" port: 34679 } }
I20260928 02:11:12.067904 11693 tablet_bootstrap.cc:492] T 339b5e8b82394f30a7fa447abd6153fb P dbac2fc64be340f2930e9c7ed30c6a22: No bootstrap required, opened a new log
I20260928 02:11:12.067616 11692 raft_consensus.cc:348] T 339b5e8b82394f30a7fa447abd6153fb P da365a6a0b7f4dd28c64233be2d6445e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "da365a6a0b7f4dd28c64233be2d6445e" member_type: VOTER last_known_addr { host: "127.6.235.67" port: 39121 } } peers { permanent_uuid: "850f1d63320043e5948f1c3ce6bc9bd5" member_type: VOTER last_known_addr { host: "127.6.235.66" port: 39725 } } peers { permanent_uuid: "dbac2fc64be340f2930e9c7ed30c6a22" member_type: VOTER last_known_addr { host: "127.6.235.65" port: 34679 } }
I20260928 02:11:12.068264 11692 raft_consensus.cc:374] T 339b5e8b82394f30a7fa447abd6153fb P da365a6a0b7f4dd28c64233be2d6445e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:11:12.068369 11693 ts_tablet_manager.cc:1397] T 339b5e8b82394f30a7fa447abd6153fb P dbac2fc64be340f2930e9c7ed30c6a22: Time spent bootstrapping tablet: real 0.018s user 0.005s sys 0.011s
I20260928 02:11:12.068445 11692 raft_consensus.cc:729] T 339b5e8b82394f30a7fa447abd6153fb P da365a6a0b7f4dd28c64233be2d6445e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: da365a6a0b7f4dd28c64233be2d6445e, State: Initialized, Role: FOLLOWER
I20260928 02:11:12.069470 11692 consensus_queue.cc:260] T 339b5e8b82394f30a7fa447abd6153fb P da365a6a0b7f4dd28c64233be2d6445e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "da365a6a0b7f4dd28c64233be2d6445e" member_type: VOTER last_known_addr { host: "127.6.235.67" port: 39121 } } peers { permanent_uuid: "850f1d63320043e5948f1c3ce6bc9bd5" member_type: VOTER last_known_addr { host: "127.6.235.66" port: 39725 } } peers { permanent_uuid: "dbac2fc64be340f2930e9c7ed30c6a22" member_type: VOTER last_known_addr { host: "127.6.235.65" port: 34679 } }
I20260928 02:11:12.076941 11672 heartbeater.cc:503] Master 127.6.235.126:44719 was elected leader, sending a full tablet report...
I20260928 02:11:12.077906 11692 ts_tablet_manager.cc:1428] T 339b5e8b82394f30a7fa447abd6153fb P da365a6a0b7f4dd28c64233be2d6445e: Time spent starting tablet: real 0.021s user 0.014s sys 0.006s
I20260928 02:11:12.078991 11539 heartbeater.cc:503] Master 127.6.235.126:44719 was elected leader, sending a full tablet report...
I20260928 02:11:12.081539 11691 ts_tablet_manager.cc:1428] T 339b5e8b82394f30a7fa447abd6153fb P 850f1d63320043e5948f1c3ce6bc9bd5: Time spent starting tablet: real 0.031s user 0.015s sys 0.011s
I20260928 02:11:12.081620 11693 raft_consensus.cc:348] T 339b5e8b82394f30a7fa447abd6153fb P dbac2fc64be340f2930e9c7ed30c6a22 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "da365a6a0b7f4dd28c64233be2d6445e" member_type: VOTER last_known_addr { host: "127.6.235.67" port: 39121 } } peers { permanent_uuid: "850f1d63320043e5948f1c3ce6bc9bd5" member_type: VOTER last_known_addr { host: "127.6.235.66" port: 39725 } } peers { permanent_uuid: "dbac2fc64be340f2930e9c7ed30c6a22" member_type: VOTER last_known_addr { host: "127.6.235.65" port: 34679 } }
I20260928 02:11:12.082226 11693 raft_consensus.cc:374] T 339b5e8b82394f30a7fa447abd6153fb P dbac2fc64be340f2930e9c7ed30c6a22 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:11:12.082386 11693 raft_consensus.cc:729] T 339b5e8b82394f30a7fa447abd6153fb P dbac2fc64be340f2930e9c7ed30c6a22 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dbac2fc64be340f2930e9c7ed30c6a22, State: Initialized, Role: FOLLOWER
I20260928 02:11:12.083062 11693 consensus_queue.cc:260] T 339b5e8b82394f30a7fa447abd6153fb P dbac2fc64be340f2930e9c7ed30c6a22 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "da365a6a0b7f4dd28c64233be2d6445e" member_type: VOTER last_known_addr { host: "127.6.235.67" port: 39121 } } peers { permanent_uuid: "850f1d63320043e5948f1c3ce6bc9bd5" member_type: VOTER last_known_addr { host: "127.6.235.66" port: 39725 } } peers { permanent_uuid: "dbac2fc64be340f2930e9c7ed30c6a22" member_type: VOTER last_known_addr { host: "127.6.235.65" port: 34679 } }
I20260928 02:11:12.085076 11693 ts_tablet_manager.cc:1428] T 339b5e8b82394f30a7fa447abd6153fb P dbac2fc64be340f2930e9c7ed30c6a22: Time spent starting tablet: real 0.017s user 0.014s sys 0.001s
W20260928 02:11:12.102505 11540 tablet.cc:2369] T 339b5e8b82394f30a7fa447abd6153fb P 850f1d63320043e5948f1c3ce6bc9bd5: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 02:11:12.141716 11673 tablet.cc:2369] T 339b5e8b82394f30a7fa447abd6153fb P da365a6a0b7f4dd28c64233be2d6445e: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 02:11:12.307853 11407 tablet.cc:2369] T 339b5e8b82394f30a7fa447abd6153fb P dbac2fc64be340f2930e9c7ed30c6a22: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 02:11:12.344947 11699 raft_consensus.cc:482] T 339b5e8b82394f30a7fa447abd6153fb P dbac2fc64be340f2930e9c7ed30c6a22 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 02:11:12.345284 11699 raft_consensus.cc:504] T 339b5e8b82394f30a7fa447abd6153fb P dbac2fc64be340f2930e9c7ed30c6a22 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "da365a6a0b7f4dd28c64233be2d6445e" member_type: VOTER last_known_addr { host: "127.6.235.67" port: 39121 } } peers { permanent_uuid: "850f1d63320043e5948f1c3ce6bc9bd5" member_type: VOTER last_known_addr { host: "127.6.235.66" port: 39725 } } peers { permanent_uuid: "dbac2fc64be340f2930e9c7ed30c6a22" member_type: VOTER last_known_addr { host: "127.6.235.65" port: 34679 } }
I20260928 02:11:12.347524 11699 leader_election.cc:288] T 339b5e8b82394f30a7fa447abd6153fb P dbac2fc64be340f2930e9c7ed30c6a22 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers da365a6a0b7f4dd28c64233be2d6445e (127.6.235.67:39121), 850f1d63320043e5948f1c3ce6bc9bd5 (127.6.235.66:39725)
I20260928 02:11:12.356148 11688 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:11:12.348158 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.235.65:35591 (local address 127.6.235.66:39725)
0928 02:11:12.348370 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0928 02:11:12.348378 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 02:11:12.350018 (+ 1640us) server_negotiation.cc:407] Connection header received
0928 02:11:12.350245 (+ 227us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:11:12.350250 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:11:12.350320 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:11:12.350421 (+ 101us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:11:12.350918 (+ 497us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:11:12.351740 (+ 822us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:11:12.353889 (+ 2149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:11:12.355135 (+ 1246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:11:12.355290 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:11:12.355371 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0928 02:11:12.355733 (+ 362us) server_negotiation.cc:299] Negotiation successful
0928 02:11:12.355870 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260928 02:11:12.356240 11701 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:11:12.348549 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.235.66:39725 (local address 127.6.235.65:35591)
0928 02:11:12.349916 (+ 1367us) negotiation.cc:107] Waiting for socket to connect
0928 02:11:12.349930 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 02:11:12.350098 (+ 168us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:11:12.350436 (+ 338us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:11:12.350440 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:11:12.350463 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:11:12.350760 (+ 297us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:11:12.350767 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:11:12.351894 (+ 1127us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:11:12.351900 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:11:12.353729 (+ 1829us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:11:12.353741 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:11:12.355352 (+ 1611us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:11:12.355358 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:11:12.355526 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:11:12.355564 (+ 38us) client_negotiation.cc:769] Sending connection context
0928 02:11:12.355705 (+ 141us) client_negotiation.cc:240] Negotiation successful
0928 02:11:12.355865 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1251,"mutex_wait_us":128,"thread_start_us":1055,"threads_started":1}
I20260928 02:11:12.356390 11687 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:11:12.348011 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.235.65:52747 (local address 127.6.235.67:39121)
0928 02:11:12.348218 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0928 02:11:12.348226 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 02:11:12.349241 (+ 1015us) server_negotiation.cc:407] Connection header received
0928 02:11:12.349513 (+ 272us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:11:12.349521 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:11:12.349605 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:11:12.349700 (+ 95us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:11:12.350517 (+ 817us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:11:12.351570 (+ 1053us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:11:12.354146 (+ 2576us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:11:12.355383 (+ 1237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:11:12.355565 (+ 182us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:11:12.355642 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0928 02:11:12.356009 (+ 367us) server_negotiation.cc:299] Negotiation successful
0928 02:11:12.356138 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260928 02:11:12.357331 11700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:11:12.347927 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.235.67:39121 (local address 127.6.235.65:52747)
0928 02:11:12.349103 (+ 1176us) negotiation.cc:107] Waiting for socket to connect
0928 02:11:12.349129 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0928 02:11:12.349322 (+ 193us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:11:12.349892 (+ 570us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:11:12.349897 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:11:12.349926 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:11:12.350332 (+ 406us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:11:12.350344 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:11:12.351795 (+ 1451us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:11:12.351800 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:11:12.353978 (+ 2178us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:11:12.353995 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:11:12.355612 (+ 1617us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:11:12.355628 (+ 16us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:11:12.355832 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:11:12.355863 (+ 31us) client_negotiation.cc:769] Sending connection context
0928 02:11:12.356791 (+ 928us) client_negotiation.cc:240] Negotiation successful
0928 02:11:12.356980 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1011,"mutex_wait_us":1,"spinlock_wait_cycles":26368,"thread_start_us":750,"threads_started":1}
I20260928 02:11:12.357752 11494 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "339b5e8b82394f30a7fa447abd6153fb" candidate_uuid: "dbac2fc64be340f2930e9c7ed30c6a22" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "850f1d63320043e5948f1c3ce6bc9bd5" is_pre_election: true
I20260928 02:11:12.358271 11627 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "339b5e8b82394f30a7fa447abd6153fb" candidate_uuid: "dbac2fc64be340f2930e9c7ed30c6a22" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "da365a6a0b7f4dd28c64233be2d6445e" is_pre_election: true
I20260928 02:11:12.358350 11494 raft_consensus.cc:2454] T 339b5e8b82394f30a7fa447abd6153fb P 850f1d63320043e5948f1c3ce6bc9bd5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dbac2fc64be340f2930e9c7ed30c6a22 in term 0.
I20260928 02:11:12.358714 11627 raft_consensus.cc:2454] T 339b5e8b82394f30a7fa447abd6153fb P da365a6a0b7f4dd28c64233be2d6445e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dbac2fc64be340f2930e9c7ed30c6a22 in term 0.
I20260928 02:11:12.359036 11294 leader_election.cc:302] T 339b5e8b82394f30a7fa447abd6153fb P dbac2fc64be340f2930e9c7ed30c6a22 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 850f1d63320043e5948f1c3ce6bc9bd5, dbac2fc64be340f2930e9c7ed30c6a22; no voters:
I20260928 02:11:12.359483 11699 raft_consensus.cc:2781] T 339b5e8b82394f30a7fa447abd6153fb P dbac2fc64be340f2930e9c7ed30c6a22 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 02:11:12.359625 11699 raft_consensus.cc:482] T 339b5e8b82394f30a7fa447abd6153fb P dbac2fc64be340f2930e9c7ed30c6a22 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 02:11:12.359704 11699 raft_consensus.cc:3037] T 339b5e8b82394f30a7fa447abd6153fb P dbac2fc64be340f2930e9c7ed30c6a22 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:11:12.362910 11699 raft_consensus.cc:504] T 339b5e8b82394f30a7fa447abd6153fb P dbac2fc64be340f2930e9c7ed30c6a22 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "da365a6a0b7f4dd28c64233be2d6445e" member_type: VOTER last_known_addr { host: "127.6.235.67" port: 39121 } } peers { permanent_uuid: "850f1d63320043e5948f1c3ce6bc9bd5" member_type: VOTER last_known_addr { host: "127.6.235.66" port: 39725 } } peers { permanent_uuid: "dbac2fc64be340f2930e9c7ed30c6a22" member_type: VOTER last_known_addr { host: "127.6.235.65" port: 34679 } }
I20260928 02:11:12.363610 11699 leader_election.cc:288] T 339b5e8b82394f30a7fa447abd6153fb P dbac2fc64be340f2930e9c7ed30c6a22 [CANDIDATE]: Term 1 election: Requested vote from peers da365a6a0b7f4dd28c64233be2d6445e (127.6.235.67:39121), 850f1d63320043e5948f1c3ce6bc9bd5 (127.6.235.66:39725)
I20260928 02:11:12.364164 11494 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "339b5e8b82394f30a7fa447abd6153fb" candidate_uuid: "dbac2fc64be340f2930e9c7ed30c6a22" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "850f1d63320043e5948f1c3ce6bc9bd5"
I20260928 02:11:12.364168 11627 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "339b5e8b82394f30a7fa447abd6153fb" candidate_uuid: "dbac2fc64be340f2930e9c7ed30c6a22" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "da365a6a0b7f4dd28c64233be2d6445e"
I20260928 02:11:12.364368 11627 raft_consensus.cc:3037] T 339b5e8b82394f30a7fa447abd6153fb P da365a6a0b7f4dd28c64233be2d6445e [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:11:12.364387 11494 raft_consensus.cc:3037] T 339b5e8b82394f30a7fa447abd6153fb P 850f1d63320043e5948f1c3ce6bc9bd5 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:11:12.366715 11494 raft_consensus.cc:2454] T 339b5e8b82394f30a7fa447abd6153fb P 850f1d63320043e5948f1c3ce6bc9bd5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dbac2fc64be340f2930e9c7ed30c6a22 in term 1.
I20260928 02:11:12.366773 11627 raft_consensus.cc:2454] T 339b5e8b82394f30a7fa447abd6153fb P da365a6a0b7f4dd28c64233be2d6445e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dbac2fc64be340f2930e9c7ed30c6a22 in term 1.
I20260928 02:11:12.367213 11294 leader_election.cc:302] T 339b5e8b82394f30a7fa447abd6153fb P dbac2fc64be340f2930e9c7ed30c6a22 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 850f1d63320043e5948f1c3ce6bc9bd5, dbac2fc64be340f2930e9c7ed30c6a22; no voters:
I20260928 02:11:12.367539 11699 raft_consensus.cc:2781] T 339b5e8b82394f30a7fa447abd6153fb P dbac2fc64be340f2930e9c7ed30c6a22 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:11:12.368014 11699 raft_consensus.cc:686] T 339b5e8b82394f30a7fa447abd6153fb P dbac2fc64be340f2930e9c7ed30c6a22 [term 1 LEADER]: Becoming Leader. State: Replica: dbac2fc64be340f2930e9c7ed30c6a22, State: Running, Role: LEADER
I20260928 02:11:12.368510 11699 consensus_queue.cc:237] T 339b5e8b82394f30a7fa447abd6153fb P dbac2fc64be340f2930e9c7ed30c6a22 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "da365a6a0b7f4dd28c64233be2d6445e" member_type: VOTER last_known_addr { host: "127.6.235.67" port: 39121 } } peers { permanent_uuid: "850f1d63320043e5948f1c3ce6bc9bd5" member_type: VOTER last_known_addr { host: "127.6.235.66" port: 39725 } } peers { permanent_uuid: "dbac2fc64be340f2930e9c7ed30c6a22" member_type: VOTER last_known_addr { host: "127.6.235.65" port: 34679 } }
I20260928 02:11:12.373646 11216 catalog_manager.cc:5495] T 339b5e8b82394f30a7fa447abd6153fb P dbac2fc64be340f2930e9c7ed30c6a22 reported cstate change: term changed from 0 to 1, leader changed from <none> to dbac2fc64be340f2930e9c7ed30c6a22 (127.6.235.65). New cstate: current_term: 1 leader_uuid: "dbac2fc64be340f2930e9c7ed30c6a22" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "da365a6a0b7f4dd28c64233be2d6445e" member_type: VOTER last_known_addr { host: "127.6.235.67" port: 39121 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "850f1d63320043e5948f1c3ce6bc9bd5" member_type: VOTER last_known_addr { host: "127.6.235.66" port: 39725 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dbac2fc64be340f2930e9c7ed30c6a22" member_type: VOTER last_known_addr { host: "127.6.235.65" port: 34679 } health_report { overall_health: HEALTHY } } }
I20260928 02:11:12.447960 11689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:11:12.442296 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42402 (local address 127.6.235.65:34679)
0928 02:11:12.442498 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0928 02:11:12.442503 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 02:11:12.442563 (+ 60us) server_negotiation.cc:407] Connection header received
0928 02:11:12.442740 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:11:12.442743 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:11:12.442821 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:11:12.442952 (+ 131us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 02:11:12.443282 (+ 330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:11:12.444012 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:11:12.445438 (+ 1426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:11:12.445951 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:11:12.446061 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:11:12.446864 (+ 803us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 02:11:12.447443 (+ 579us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 02:11:12.447559 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0928 02:11:12.447677 (+ 118us) server_negotiation.cc:299] Negotiation successful
0928 02:11:12.447748 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260928 02:11:12.456463 11494 raft_consensus.cc:1261] T 339b5e8b82394f30a7fa447abd6153fb P 850f1d63320043e5948f1c3ce6bc9bd5 [term 1 FOLLOWER]: Refusing update from remote peer dbac2fc64be340f2930e9c7ed30c6a22: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 02:11:12.456482 11627 raft_consensus.cc:1261] T 339b5e8b82394f30a7fa447abd6153fb P da365a6a0b7f4dd28c64233be2d6445e [term 1 FOLLOWER]: Refusing update from remote peer dbac2fc64be340f2930e9c7ed30c6a22: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 02:11:12.457616 11702 consensus_queue.cc:1035] T 339b5e8b82394f30a7fa447abd6153fb P dbac2fc64be340f2930e9c7ed30c6a22 [LEADER]: Connected to new peer: Peer: permanent_uuid: "850f1d63320043e5948f1c3ce6bc9bd5" member_type: VOTER last_known_addr { host: "127.6.235.66" port: 39725 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 02:11:12.457995 11699 consensus_queue.cc:1035] T 339b5e8b82394f30a7fa447abd6153fb P dbac2fc64be340f2930e9c7ed30c6a22 [LEADER]: Connected to new peer: Peer: permanent_uuid: "da365a6a0b7f4dd28c64233be2d6445e" member_type: VOTER last_known_addr { host: "127.6.235.67" port: 39121 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 02:11:12.471159 11706 mvcc.cc:205] Tried to move back new op lower bound from 7334139791171526656 to 7334139790825332736. Current Snapshot: MvccSnapshot[applied={T|T < 7334139791171526656}]
I20260928 02:11:12.472159 11705 mvcc.cc:205] Tried to move back new op lower bound from 7334139791171526656 to 7334139790825332736. Current Snapshot: MvccSnapshot[applied={T|T < 7334139791171526656}]
I20260928 02:11:12.473435 11707 mvcc.cc:205] Tried to move back new op lower bound from 7334139791171526656 to 7334139790825332736. Current Snapshot: MvccSnapshot[applied={T|T < 7334139791171526656}]
I20260928 02:11:12.486312 11216 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35044:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 02:11:12.486631 11216 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35044:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 02:11:12.490634 11216 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 5b6f10a43e254df199f2ddcd91f77790: Sending DeleteTablet for 3 replicas of tablet 339b5e8b82394f30a7fa447abd6153fb
I20260928 02:11:12.491953 11607 tablet_service.cc:1514] Processing DeleteTablet for tablet 339b5e8b82394f30a7fa447abd6153fb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:11:12 UTC) from {username='kudu', principal='kudu/127.6.235.126@KRBTEST.COM'} at 127.0.0.1:55740
I20260928 02:11:12.492004 11341 tablet_service.cc:1514] Processing DeleteTablet for tablet 339b5e8b82394f30a7fa447abd6153fb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:11:12 UTC) from {username='kudu', principal='kudu/127.6.235.126@KRBTEST.COM'} at 127.0.0.1:42390
I20260928 02:11:12.493587 11474 tablet_service.cc:1514] Processing DeleteTablet for tablet 339b5e8b82394f30a7fa447abd6153fb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:11:12 UTC) from {username='kudu', principal='kudu/127.6.235.126@KRBTEST.COM'} at 127.0.0.1:33698
I20260928 02:11:12.494742 7085 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskTlaUuc/build/asan/bin/kudu pid 11278
I20260928 02:11:12.506135 11715 tablet_replica.cc:330] T 339b5e8b82394f30a7fa447abd6153fb P da365a6a0b7f4dd28c64233be2d6445e: stopping tablet replica
I20260928 02:11:12.506776 11715 raft_consensus.cc:2229] T 339b5e8b82394f30a7fa447abd6153fb P da365a6a0b7f4dd28c64233be2d6445e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 02:11:12.507476 11715 raft_consensus.cc:2258] T 339b5e8b82394f30a7fa447abd6153fb P da365a6a0b7f4dd28c64233be2d6445e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 02:11:12.511591 11715 ts_tablet_manager.cc:1905] T 339b5e8b82394f30a7fa447abd6153fb P da365a6a0b7f4dd28c64233be2d6445e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 02:11:12.519224 11715 ts_tablet_manager.cc:1918] T 339b5e8b82394f30a7fa447abd6153fb P da365a6a0b7f4dd28c64233be2d6445e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 02:11:12.519526 11715 log.cc:1196] T 339b5e8b82394f30a7fa447abd6153fb P da365a6a0b7f4dd28c64233be2d6445e: Deleting WAL directory at /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790561432251112-7085-0/minicluster-data/ts-2/wal/wals/339b5e8b82394f30a7fa447abd6153fb
I20260928 02:11:12.520169 11715 ts_tablet_manager.cc:1939] T 339b5e8b82394f30a7fa447abd6153fb P da365a6a0b7f4dd28c64233be2d6445e: Deleting consensus metadata
I20260928 02:11:12.522352 11203 catalog_manager.cc:4841] TS da365a6a0b7f4dd28c64233be2d6445e (127.6.235.67:39121): tablet 339b5e8b82394f30a7fa447abd6153fb (table test-table [id=963beae81a17432fbf6f6a1fbf70e924]) successfully deleted
I20260928 02:11:12.523665 11717 tablet_replica.cc:330] T 339b5e8b82394f30a7fa447abd6153fb P 850f1d63320043e5948f1c3ce6bc9bd5: stopping tablet replica
I20260928 02:11:12.524426 11717 raft_consensus.cc:2229] T 339b5e8b82394f30a7fa447abd6153fb P 850f1d63320043e5948f1c3ce6bc9bd5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 02:11:12.525259 11717 raft_consensus.cc:2258] T 339b5e8b82394f30a7fa447abd6153fb P 850f1d63320043e5948f1c3ce6bc9bd5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 02:11:12.527038 11716 tablet_replica.cc:330] T 339b5e8b82394f30a7fa447abd6153fb P dbac2fc64be340f2930e9c7ed30c6a22: stopping tablet replica
I20260928 02:11:12.527442 11716 raft_consensus.cc:2229] T 339b5e8b82394f30a7fa447abd6153fb P dbac2fc64be340f2930e9c7ed30c6a22 [term 1 LEADER]: Raft consensus shutting down.
I20260928 02:11:12.528087 11716 raft_consensus.cc:2258] T 339b5e8b82394f30a7fa447abd6153fb P dbac2fc64be340f2930e9c7ed30c6a22 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 02:11:12.529675 11717 ts_tablet_manager.cc:1905] T 339b5e8b82394f30a7fa447abd6153fb P 850f1d63320043e5948f1c3ce6bc9bd5: Deleting tablet data with delete state TABLET_DATA_DELETED
Sep 28 02:11:12 dist-test-slave-86gv krb5kdc[11173](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561469, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.235.65@KRBTEST.COM
I20260928 02:11:12.535648 11717 ts_tablet_manager.cc:1918] T 339b5e8b82394f30a7fa447abd6153fb P 850f1d63320043e5948f1c3ce6bc9bd5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 02:11:12.535876 11717 log.cc:1196] T 339b5e8b82394f30a7fa447abd6153fb P 850f1d63320043e5948f1c3ce6bc9bd5: Deleting WAL directory at /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790561432251112-7085-0/minicluster-data/ts-1/wal/wals/339b5e8b82394f30a7fa447abd6153fb
I20260928 02:11:12.536458 11717 ts_tablet_manager.cc:1939] T 339b5e8b82394f30a7fa447abd6153fb P 850f1d63320043e5948f1c3ce6bc9bd5: Deleting consensus metadata
I20260928 02:11:12.537690 11716 ts_tablet_manager.cc:1905] T 339b5e8b82394f30a7fa447abd6153fb P dbac2fc64be340f2930e9c7ed30c6a22: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 02:11:12.538405 11202 catalog_manager.cc:4841] TS 850f1d63320043e5948f1c3ce6bc9bd5 (127.6.235.66:39725): tablet 339b5e8b82394f30a7fa447abd6153fb (table test-table [id=963beae81a17432fbf6f6a1fbf70e924]) successfully deleted
I20260928 02:11:12.543092 11689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:11:12.496389 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42414 (local address 127.6.235.65:34679)
0928 02:11:12.496614 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0928 02:11:12.496623 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 02:11:12.521710 (+ 25087us) server_negotiation.cc:407] Connection header received
0928 02:11:12.523128 (+ 1418us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:11:12.523134 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:11:12.523233 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:11:12.523373 (+ 140us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:11:12.525298 (+ 1925us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:11:12.526587 (+ 1289us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:11:12.528601 (+ 2014us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:11:12.529325 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:11:12.529433 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:11:12.534973 (+ 5540us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:11:12.534991 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:11:12.535030 (+ 39us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:11:12.535085 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:11:12.539505 (+ 4420us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:11:12.540500 (+ 995us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:11:12.540511 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:11:12.540515 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:11:12.540708 (+ 193us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:11:12.541218 (+ 510us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:11:12.541224 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:11:12.541227 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:11:12.541711 (+ 484us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:11:12.542112 (+ 401us) server_negotiation.cc:1035] Waiting for connection context
0928 02:11:12.542398 (+ 286us) server_negotiation.cc:299] Negotiation successful
0928 02:11:12.542682 (+ 284us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260928 02:11:12.543934 11401 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 02:11:12.647840 11716 ts_tablet_manager.cc:1918] T 339b5e8b82394f30a7fa447abd6153fb P dbac2fc64be340f2930e9c7ed30c6a22: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 02:11:12.648408 11716 log.cc:1196] T 339b5e8b82394f30a7fa447abd6153fb P dbac2fc64be340f2930e9c7ed30c6a22: Deleting WAL directory at /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790561432251112-7085-0/minicluster-data/ts-0/wal/wals/339b5e8b82394f30a7fa447abd6153fb
I20260928 02:11:12.649106 11716 ts_tablet_manager.cc:1939] T 339b5e8b82394f30a7fa447abd6153fb P dbac2fc64be340f2930e9c7ed30c6a22: Deleting consensus metadata
I20260928 02:11:12.649873 7085 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTlaUuc/build/asan/bin/kudu with pid 11278
W20260928 02:11:12.667801 11202 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.6.235.65:34679 (error 108)
I20260928 02:11:12.668109 7085 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskTlaUuc/build/asan/bin/kudu pid 11412
I20260928 02:11:12.669930 11721 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:11:12.669151 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.235.65:34679 (local address 127.0.0.1:42422)
0928 02:11:12.669626 (+ 475us) negotiation.cc:107] Waiting for socket to connect
0928 02:11:12.669750 (+ 124us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.235.65:34679: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":329,"thread_start_us":245,"threads_started":1}
W20260928 02:11:12.670176 11202 catalog_manager.cc:4584] TS dbac2fc64be340f2930e9c7ed30c6a22 (127.6.235.65:34679): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 339b5e8b82394f30a7fa447abd6153fb: Network error: Client connection negotiation failed: client connection to 127.6.235.65:34679: connect: Connection refused (error 111)
Sep 28 02:11:12 dist-test-slave-86gv krb5kdc[11173](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561469, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.235.66@KRBTEST.COM
I20260928 02:11:12.684437 11688 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:11:12.668987 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33710 (local address 127.6.235.66:39725)
0928 02:11:12.669172 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0928 02:11:12.669178 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 02:11:12.669289 (+ 111us) server_negotiation.cc:407] Connection header received
0928 02:11:12.669512 (+ 223us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:11:12.669518 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:11:12.669600 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:11:12.669723 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:11:12.671088 (+ 1365us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:11:12.671781 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:11:12.673199 (+ 1418us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:11:12.673634 (+ 435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:11:12.673775 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:11:12.678108 (+ 4333us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:11:12.678124 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:11:12.678140 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:11:12.678189 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:11:12.681424 (+ 3235us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:11:12.682180 (+ 756us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:11:12.682188 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:11:12.682191 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:11:12.682332 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:11:12.682810 (+ 478us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:11:12.682815 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:11:12.682817 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:11:12.683210 (+ 393us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:11:12.683511 (+ 301us) server_negotiation.cc:1035] Waiting for connection context
0928 02:11:12.683859 (+ 348us) server_negotiation.cc:299] Negotiation successful
0928 02:11:12.684079 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260928 02:11:12.685135 11534 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 02:11:12.723857 11721 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:11:12.723364 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.235.65:34679 (local address 127.0.0.1:42430)
0928 02:11:12.723573 (+ 209us) negotiation.cc:107] Waiting for socket to connect
0928 02:11:12.723662 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.235.65:34679: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":77}
I20260928 02:11:12.725032 11721 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:11:12.724612 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.235.65:34679 (local address 127.0.0.1:42440)
0928 02:11:12.724740 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0928 02:11:12.724808 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.235.65:34679: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
I20260928 02:11:12.775717 7085 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTlaUuc/build/asan/bin/kudu with pid 11412
I20260928 02:11:12.785758 11721 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:11:12.785174 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.235.65:34679 (local address 127.0.0.1:42456)
0928 02:11:12.785415 (+ 241us) negotiation.cc:107] Waiting for socket to connect
0928 02:11:12.785540 (+ 125us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.235.65:34679: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
I20260928 02:11:12.786998 11721 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:11:12.786557 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.235.65:34679 (local address 127.0.0.1:42468)
0928 02:11:12.786709 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0928 02:11:12.786790 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.235.65:34679: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
I20260928 02:11:12.794266 7085 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskTlaUuc/build/asan/bin/kudu pid 11545
Sep 28 02:11:12 dist-test-slave-86gv krb5kdc[11173](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561469, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.235.67@KRBTEST.COM
I20260928 02:11:12.809386 11687 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:11:12.795090 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55748 (local address 127.6.235.67:39121)
0928 02:11:12.795271 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0928 02:11:12.795279 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 02:11:12.795335 (+ 56us) server_negotiation.cc:407] Connection header received
0928 02:11:12.795519 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:11:12.795523 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:11:12.795590 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:11:12.795694 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:11:12.796948 (+ 1254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:11:12.797659 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:11:12.798887 (+ 1228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:11:12.799312 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:11:12.799448 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:11:12.803805 (+ 4357us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:11:12.803820 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:11:12.803836 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:11:12.803877 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:11:12.806682 (+ 2805us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:11:12.807336 (+ 654us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:11:12.807344 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:11:12.807346 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:11:12.807483 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:11:12.807857 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:11:12.807864 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:11:12.807866 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:11:12.808215 (+ 349us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:11:12.808528 (+ 313us) server_negotiation.cc:1035] Waiting for connection context
0928 02:11:12.808806 (+ 278us) server_negotiation.cc:299] Negotiation successful
0928 02:11:12.809033 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260928 02:11:12.810135 11667 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 02:11:12.867553 11721 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:11:12.866958 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.235.65:34679 (local address 127.0.0.1:42480)
0928 02:11:12.867261 (+ 303us) negotiation.cc:107] Waiting for socket to connect
0928 02:11:12.867375 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.235.65:34679: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":111}
I20260928 02:11:12.868736 11721 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:11:12.868275 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.235.65:34679 (local address 127.0.0.1:42484)
0928 02:11:12.868482 (+ 207us) negotiation.cc:107] Waiting for socket to connect
0928 02:11:12.868575 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.235.65:34679: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":107}
I20260928 02:11:12.900235 7085 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTlaUuc/build/asan/bin/kudu with pid 11545
I20260928 02:11:12.917753 7085 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskTlaUuc/build/asan/bin/kudu pid 11186
I20260928 02:11:12.918967 11247 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 02:11:13.027278 7085 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTlaUuc/build/asan/bin/kudu with pid 11186
2026-09-28T02:11:13Z chronyd exiting
[ OK ] SecurityITest.TestDisableWebUI (4659 ms)
[ RUN ] SecurityITest.TestDisableAuthenticationEncryption
2026-09-28T02:11:13Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T02:11:13Z Disabled control of system clock
I20260928 02:11:13.098242 7085 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTlaUuc/build/asan/bin/kudu
/tmp/dist-test-taskTlaUuc/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790561432251112-7085-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790561432251112-7085-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790561432251112-7085-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790561432251112-7085-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.235.126:45407
--webserver_interface=127.6.235.126
--webserver_port=0
--builtin_ntp_servers=127.6.235.84:42613
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.235.126:45407
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260928 02:11:13.436627 11728 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:11:13.437075 11728 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:11:13.437217 11728 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:11:13.444656 11728 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 02:11:13.444775 11728 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:11:13.444866 11728 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:11:13.444990 11728 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 02:11:13.445076 11728 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 02:11:13.454718 11728 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.235.84:42613
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790561432251112-7085-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790561432251112-7085-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.235.126:45407
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.235.126:45407
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790561432251112-7085-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.235.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790561432251112-7085-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 00:51:59 UTC on ccycloud.quasar-axgkwn.root.comops.site
build id 4676464
ASAN enabled
I20260928 02:11:13.456120 11728 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:11:13.458141 11728 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:11:13.470201 11733 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:11:13.470240 11734 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:11:13.470276 11736 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:11:13.471019 11728 server_base.cc:1034] running on GCE node
I20260928 02:11:13.471850 11728 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:11:13.473501 11728 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:11:13.474746 11728 hybrid_clock.cc:648] HybridClock initialized: now 1790561473474702 us; error 51 us; skew 500 ppm
I20260928 02:11:13.475090 11728 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260928 02:11:13.477490 11728 webserver.cc:460] Webserver started at http://127.6.235.126:43963/ using document root <none> and password file <none>
I20260928 02:11:13.478204 11728 fs_manager.cc:359] Metadata directory not provided
I20260928 02:11:13.478330 11728 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:11:13.478722 11728 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:11:13.481773 11728 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790561432251112-7085-0/minicluster-data/master-0/data/instance:
uuid: "a19969de5aa54329851490c5e575daa7"
format_stamp: "Formatted at 2026-09-28 02:11:13 on dist-test-slave-86gv"
server_key: "48bbe57b09c05f0a92bfbd535b303e1a"
server_key_iv: "dab55e5e99a2fa58390cfcd8392d8657"
server_key_version: "encryptionkey@0"
I20260928 02:11:13.482649 11728 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790561432251112-7085-0/minicluster-data/master-0/wal/instance:
uuid: "a19969de5aa54329851490c5e575daa7"
format_stamp: "Formatted at 2026-09-28 02:11:13 on dist-test-slave-86gv"
server_key: "48bbe57b09c05f0a92bfbd535b303e1a"
server_key_iv: "dab55e5e99a2fa58390cfcd8392d8657"
server_key_version: "encryptionkey@0"
I20260928 02:11:13.487805 11728 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260928 02:11:13.491182 11742 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:11:13.492506 11728 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260928 02:11:13.492676 11728 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790561432251112-7085-0/minicluster-data/master-0/data,/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790561432251112-7085-0/minicluster-data/master-0/wal
uuid: "a19969de5aa54329851490c5e575daa7"
format_stamp: "Formatted at 2026-09-28 02:11:13 on dist-test-slave-86gv"
server_key: "48bbe57b09c05f0a92bfbd535b303e1a"
server_key_iv: "dab55e5e99a2fa58390cfcd8392d8657"
server_key_version: "encryptionkey@0"
I20260928 02:11:13.492931 11728 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790561432251112-7085-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790561432251112-7085-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790561432251112-7085-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:11:13.498741 11728 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:11:13.499529 11728 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:11:13.499826 11728 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:11:13.518304 11728 rpc_server.cc:307] RPC server started. Bound to: 127.6.235.126:45407
I20260928 02:11:13.518361 11793 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.235.126:45407 every 8 connection(s)
I20260928 02:11:13.519695 11728 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790561432251112-7085-0/minicluster-data/master-0/data/info.pb
I20260928 02:11:13.523489 11794 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:11:13.528649 7085 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTlaUuc/build/asan/bin/kudu as pid 11728
I20260928 02:11:13.528959 7085 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790561432251112-7085-0/minicluster-data/master-0/wal/instance
I20260928 02:11:13.529594 7085 external_mini_cluster.cc:1428] Setting key 6291cf5123ea7520b8959779711a1430
I20260928 02:11:13.534387 11794 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a19969de5aa54329851490c5e575daa7: Bootstrap starting.
I20260928 02:11:13.538059 11794 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a19969de5aa54329851490c5e575daa7: Neither blocks nor log segments found. Creating new log.
I20260928 02:11:13.538440 11797 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:11:13.531820 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60960 (local address 127.6.235.126:45407)
0928 02:11:13.532990 (+ 1170us) server_negotiation.cc:206] Beginning negotiation
0928 02:11:13.533002 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0928 02:11:13.533046 (+ 44us) server_negotiation.cc:407] Connection header received
0928 02:11:13.534744 (+ 1698us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:11:13.534766 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:11:13.535199 (+ 433us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:11:13.535528 (+ 329us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:11:13.536754 (+ 1226us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:11:13.536761 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:11:13.536775 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 02:11:13.536789 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:11:13.536855 (+ 66us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 02:11:13.536872 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:11:13.537100 (+ 228us) server_negotiation.cc:1035] Waiting for connection context
0928 02:11:13.537200 (+ 100us) server_negotiation.cc:299] Negotiation successful
0928 02:11:13.537466 (+ 266us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":640,"thread_start_us":402,"threads_started":1}
I20260928 02:11:13.539237 11794 log.cc:824] T 00000000000000000000000000000000 P a19969de5aa54329851490c5e575daa7: Log is configured to *not* fsync() on all Append() calls
I20260928 02:11:13.542237 11794 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a19969de5aa54329851490c5e575daa7: No bootstrap required, opened a new log
I20260928 02:11:13.550398 11794 raft_consensus.cc:348] T 00000000000000000000000000000000 P a19969de5aa54329851490c5e575daa7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a19969de5aa54329851490c5e575daa7" member_type: VOTER last_known_addr { host: "127.6.235.126" port: 45407 } }
I20260928 02:11:13.550832 11794 raft_consensus.cc:374] T 00000000000000000000000000000000 P a19969de5aa54329851490c5e575daa7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:11:13.550948 11794 raft_consensus.cc:729] T 00000000000000000000000000000000 P a19969de5aa54329851490c5e575daa7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a19969de5aa54329851490c5e575daa7, State: Initialized, Role: FOLLOWER
I20260928 02:11:13.551606 11794 consensus_queue.cc:260] T 00000000000000000000000000000000 P a19969de5aa54329851490c5e575daa7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a19969de5aa54329851490c5e575daa7" member_type: VOTER last_known_addr { host: "127.6.235.126" port: 45407 } }
I20260928 02:11:13.551863 11794 raft_consensus.cc:388] T 00000000000000000000000000000000 P a19969de5aa54329851490c5e575daa7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 02:11:13.552008 11794 raft_consensus.cc:482] T 00000000000000000000000000000000 P a19969de5aa54329851490c5e575daa7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 02:11:13.552189 11794 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a19969de5aa54329851490c5e575daa7 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:11:13.554540 11794 raft_consensus.cc:504] T 00000000000000000000000000000000 P a19969de5aa54329851490c5e575daa7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a19969de5aa54329851490c5e575daa7" member_type: VOTER last_known_addr { host: "127.6.235.126" port: 45407 } }
I20260928 02:11:13.555171 11794 leader_election.cc:302] T 00000000000000000000000000000000 P a19969de5aa54329851490c5e575daa7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a19969de5aa54329851490c5e575daa7; no voters:
I20260928 02:11:13.556102 11794 leader_election.cc:288] T 00000000000000000000000000000000 P a19969de5aa54329851490c5e575daa7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 02:11:13.556314 11799 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a19969de5aa54329851490c5e575daa7 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:11:13.557379 11799 raft_consensus.cc:686] T 00000000000000000000000000000000 P a19969de5aa54329851490c5e575daa7 [term 1 LEADER]: Becoming Leader. State: Replica: a19969de5aa54329851490c5e575daa7, State: Running, Role: LEADER
I20260928 02:11:13.557953 11799 consensus_queue.cc:237] T 00000000000000000000000000000000 P a19969de5aa54329851490c5e575daa7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a19969de5aa54329851490c5e575daa7" member_type: VOTER last_known_addr { host: "127.6.235.126" port: 45407 } }
I20260928 02:11:13.558282 11794 sys_catalog.cc:564] T 00000000000000000000000000000000 P a19969de5aa54329851490c5e575daa7 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 02:11:13.560812 11801 sys_catalog.cc:455] T 00000000000000000000000000000000 P a19969de5aa54329851490c5e575daa7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a19969de5aa54329851490c5e575daa7. Latest consensus state: current_term: 1 leader_uuid: "a19969de5aa54329851490c5e575daa7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a19969de5aa54329851490c5e575daa7" member_type: VOTER last_known_addr { host: "127.6.235.126" port: 45407 } } }
I20260928 02:11:13.561211 11801 sys_catalog.cc:458] T 00000000000000000000000000000000 P a19969de5aa54329851490c5e575daa7 [sys.catalog]: This master's current role is: LEADER
I20260928 02:11:13.560936 11800 sys_catalog.cc:455] T 00000000000000000000000000000000 P a19969de5aa54329851490c5e575daa7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a19969de5aa54329851490c5e575daa7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a19969de5aa54329851490c5e575daa7" member_type: VOTER last_known_addr { host: "127.6.235.126" port: 45407 } } }
I20260928 02:11:13.561368 11800 sys_catalog.cc:458] T 00000000000000000000000000000000 P a19969de5aa54329851490c5e575daa7 [sys.catalog]: This master's current role is: LEADER
I20260928 02:11:13.563401 11805 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 02:11:13.568836 11805 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 02:11:13.578748 11805 catalog_manager.cc:1347] Generated new cluster ID: ffaddde8c37648e19cc9980a1259a2e5
I20260928 02:11:13.578895 11805 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 02:11:13.615088 11805 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 02:11:13.615572 11805 catalog_manager.cc:1504] Loading token signing keys...
I20260928 02:11:13.624027 11805 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a19969de5aa54329851490c5e575daa7: Generated new TSK 0
I20260928 02:11:13.625145 11805 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 02:11:13.638129 7085 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTlaUuc/build/asan/bin/kudu
/tmp/dist-test-taskTlaUuc/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790561432251112-7085-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790561432251112-7085-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790561432251112-7085-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790561432251112-7085-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.235.65:0
--local_ip_for_outbound_sockets=127.6.235.65
--webserver_interface=127.6.235.65
--webserver_port=0
--tserver_master_addrs=127.6.235.126:45407
--builtin_ntp_servers=127.6.235.84:42613
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260928 02:11:13.975649 11818 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:11:13.975950 11818 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:11:13.976105 11818 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:11:13.983759 11818 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:11:13.983878 11818 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:11:13.984061 11818 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.235.65
I20260928 02:11:13.993712 11818 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.235.84:42613
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790561432251112-7085-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790561432251112-7085-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.235.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790561432251112-7085-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.6.235.65
--webserver_port=0
--tserver_master_addrs=127.6.235.126:45407
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.235.65
--log_dir=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790561432251112-7085-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 00:51:59 UTC on ccycloud.quasar-axgkwn.root.comops.site
build id 4676464
ASAN enabled
I20260928 02:11:13.995131 11818 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:11:13.997244 11818 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:11:14.011561 11824 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:11:14.011658 11826 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:11:14.011591 11823 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:11:14.012384 11818 server_base.cc:1034] running on GCE node
I20260928 02:11:14.013108 11818 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:11:14.014130 11818 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:11:14.015385 11818 hybrid_clock.cc:648] HybridClock initialized: now 1790561474015302 us; error 46 us; skew 500 ppm
I20260928 02:11:14.015738 11818 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260928 02:11:14.018123 11818 webserver.cc:460] Webserver started at http://127.6.235.65:34969/ using document root <none> and password file <none>
I20260928 02:11:14.018857 11818 fs_manager.cc:359] Metadata directory not provided
I20260928 02:11:14.018977 11818 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:11:14.019317 11818 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:11:14.022430 11818 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790561432251112-7085-0/minicluster-data/ts-0/data/instance:
uuid: "0b983ff76977406fb78575b56e369e6c"
format_stamp: "Formatted at 2026-09-28 02:11:14 on dist-test-slave-86gv"
server_key: "3c42acb039f0448ec077c3abeef5b28e"
server_key_iv: "d2357d8f94cc74cbd39d089f9abceac6"
server_key_version: "encryptionkey@0"
I20260928 02:11:14.023372 11818 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790561432251112-7085-0/minicluster-data/ts-0/wal/instance:
uuid: "0b983ff76977406fb78575b56e369e6c"
format_stamp: "Formatted at 2026-09-28 02:11:14 on dist-test-slave-86gv"
server_key: "3c42acb039f0448ec077c3abeef5b28e"
server_key_iv: "d2357d8f94cc74cbd39d089f9abceac6"
server_key_version: "encryptionkey@0"
I20260928 02:11:14.029055 11818 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260928 02:11:14.032809 11832 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:11:14.034384 11818 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 02:11:14.034572 11818 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790561432251112-7085-0/minicluster-data/ts-0/data,/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790561432251112-7085-0/minicluster-data/ts-0/wal
uuid: "0b983ff76977406fb78575b56e369e6c"
format_stamp: "Formatted at 2026-09-28 02:11:14 on dist-test-slave-86gv"
server_key: "3c42acb039f0448ec077c3abeef5b28e"
server_key_iv: "d2357d8f94cc74cbd39d089f9abceac6"
server_key_version: "encryptionkey@0"
I20260928 02:11:14.034842 11818 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790561432251112-7085-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790561432251112-7085-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790561432251112-7085-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:11:14.041363 11818 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:11:14.042259 11818 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:11:14.042554 11818 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:11:14.043582 11818 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:11:14.045346 11818 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:11:14.045440 11818 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:11:14.045594 11818 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:11:14.045692 11818 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:11:14.081307 11818 rpc_server.cc:307] RPC server started. Bound to: 127.6.235.65:34135
I20260928 02:11:14.081364 11944 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.235.65:34135 every 8 connection(s)
I20260928 02:11:14.082791 11818 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790561432251112-7085-0/minicluster-data/ts-0/data/info.pb
I20260928 02:11:14.089046 7085 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTlaUuc/build/asan/bin/kudu as pid 11818
I20260928 02:11:14.089267 7085 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790561432251112-7085-0/minicluster-data/ts-0/wal/instance
I20260928 02:11:14.090080 7085 external_mini_cluster.cc:1428] Setting key 1668869a13da6ea4ea5de981c4df98a4
I20260928 02:11:14.092259 11947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:11:14.085683 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.235.65:33109 (local address 127.6.235.126:45407)
0928 02:11:14.086157 (+ 474us) server_negotiation.cc:206] Beginning negotiation
0928 02:11:14.086163 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 02:11:14.087521 (+ 1358us) server_negotiation.cc:407] Connection header received
0928 02:11:14.089527 (+ 2006us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:11:14.089532 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:11:14.089591 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:11:14.089719 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:11:14.091422 (+ 1703us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:11:14.091429 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:11:14.091436 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 02:11:14.091459 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:11:14.091501 (+ 42us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 02:11:14.091514 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:11:14.091659 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0928 02:11:14.091878 (+ 219us) server_negotiation.cc:299] Negotiation successful
0928 02:11:14.091948 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":346,"thread_start_us":226,"threads_started":1}
I20260928 02:11:14.093003 7085 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTlaUuc/build/asan/bin/kudu
/tmp/dist-test-taskTlaUuc/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790561432251112-7085-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790561432251112-7085-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790561432251112-7085-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790561432251112-7085-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.235.66:0
--local_ip_for_outbound_sockets=127.6.235.66
--webserver_interface=127.6.235.66
--webserver_port=0
--tserver_master_addrs=127.6.235.126:45407
--builtin_ntp_servers=127.6.235.84:42613
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260928 02:11:14.093501 11948 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:11:14.086407 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.235.126:45407 (local address 127.6.235.65:33109)
0928 02:11:14.087342 (+ 935us) negotiation.cc:107] Waiting for socket to connect
0928 02:11:14.087387 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0928 02:11:14.089187 (+ 1800us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:11:14.090036 (+ 849us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:11:14.090049 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:11:14.090193 (+ 144us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:11:14.091032 (+ 839us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 02:11:14.091034 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:11:14.091078 (+ 44us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 02:11:14.091232 (+ 154us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:11:14.091627 (+ 395us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:11:14.091638 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:11:14.091647 (+ 9us) client_negotiation.cc:769] Sending connection context
0928 02:11:14.091958 (+ 311us) client_negotiation.cc:240] Negotiation successful
0928 02:11:14.092274 (+ 316us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":484,"thread_start_us":268,"threads_started":1}
I20260928 02:11:14.095335 11945 heartbeater.cc:347] Connected to a master server at 127.6.235.126:45407
I20260928 02:11:14.095698 11945 heartbeater.cc:464] Registering TS with master...
I20260928 02:11:14.096318 11945 heartbeater.cc:511] Master 127.6.235.126:45407 requested a full tablet report, sending...
I20260928 02:11:14.098835 11759 ts_manager.cc:194] Registered new tserver with Master: 0b983ff76977406fb78575b56e369e6c (127.6.235.65:34135)
W20260928 02:11:14.442037 11949 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:11:14.442337 11949 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:11:14.442483 11949 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:11:14.449856 11949 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:11:14.449962 11949 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:11:14.450152 11949 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.235.66
I20260928 02:11:14.459640 11949 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.235.84:42613
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790561432251112-7085-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790561432251112-7085-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.235.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790561432251112-7085-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.6.235.66
--webserver_port=0
--tserver_master_addrs=127.6.235.126:45407
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.235.66
--log_dir=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790561432251112-7085-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 00:51:59 UTC on ccycloud.quasar-axgkwn.root.comops.site
build id 4676464
ASAN enabled
I20260928 02:11:14.461004 11949 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:11:14.462808 11949 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:11:14.476455 11957 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:11:14.476585 11955 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:11:14.476610 11954 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:11:14.477532 11949 server_base.cc:1034] running on GCE node
I20260928 02:11:14.478181 11949 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:11:14.479163 11949 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:11:14.480406 11949 hybrid_clock.cc:648] HybridClock initialized: now 1790561474480364 us; error 63 us; skew 500 ppm
I20260928 02:11:14.480751 11949 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260928 02:11:14.483057 11949 webserver.cc:460] Webserver started at http://127.6.235.66:39111/ using document root <none> and password file <none>
I20260928 02:11:14.483757 11949 fs_manager.cc:359] Metadata directory not provided
I20260928 02:11:14.483886 11949 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:11:14.484210 11949 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:11:14.487099 11949 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790561432251112-7085-0/minicluster-data/ts-1/data/instance:
uuid: "ca4d5f9b932a477fa95363975fb8a511"
format_stamp: "Formatted at 2026-09-28 02:11:14 on dist-test-slave-86gv"
server_key: "328b59dfec3d06a2d296cd81aead0329"
server_key_iv: "9dff6b7d210410f17d59a5c532d6a3a9"
server_key_version: "encryptionkey@0"
I20260928 02:11:14.487892 11949 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790561432251112-7085-0/minicluster-data/ts-1/wal/instance:
uuid: "ca4d5f9b932a477fa95363975fb8a511"
format_stamp: "Formatted at 2026-09-28 02:11:14 on dist-test-slave-86gv"
server_key: "328b59dfec3d06a2d296cd81aead0329"
server_key_iv: "9dff6b7d210410f17d59a5c532d6a3a9"
server_key_version: "encryptionkey@0"
I20260928 02:11:14.493388 11949 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.004s
I20260928 02:11:14.497303 11963 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:11:14.499106 11949 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260928 02:11:14.499360 11949 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790561432251112-7085-0/minicluster-data/ts-1/data,/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790561432251112-7085-0/minicluster-data/ts-1/wal
uuid: "ca4d5f9b932a477fa95363975fb8a511"
format_stamp: "Formatted at 2026-09-28 02:11:14 on dist-test-slave-86gv"
server_key: "328b59dfec3d06a2d296cd81aead0329"
server_key_iv: "9dff6b7d210410f17d59a5c532d6a3a9"
server_key_version: "encryptionkey@0"
I20260928 02:11:14.499627 11949 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790561432251112-7085-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790561432251112-7085-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790561432251112-7085-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:11:14.506354 11949 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:11:14.507452 11949 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:11:14.507790 11949 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:11:14.508823 11949 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:11:14.510537 11949 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:11:14.510671 11949 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:11:14.510833 11949 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:11:14.510929 11949 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:11:14.550712 11949 rpc_server.cc:307] RPC server started. Bound to: 127.6.235.66:34425
I20260928 02:11:14.550748 12075 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.235.66:34425 every 8 connection(s)
I20260928 02:11:14.552225 11949 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790561432251112-7085-0/minicluster-data/ts-1/data/info.pb
I20260928 02:11:14.556545 7085 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTlaUuc/build/asan/bin/kudu as pid 11949
I20260928 02:11:14.556735 7085 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790561432251112-7085-0/minicluster-data/ts-1/wal/instance
I20260928 02:11:14.557372 7085 external_mini_cluster.cc:1428] Setting key 18a173f5c6172c88f8bce7ab84872903
I20260928 02:11:14.560653 7085 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTlaUuc/build/asan/bin/kudu
/tmp/dist-test-taskTlaUuc/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790561432251112-7085-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790561432251112-7085-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790561432251112-7085-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790561432251112-7085-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.235.67:0
--local_ip_for_outbound_sockets=127.6.235.67
--webserver_interface=127.6.235.67
--webserver_port=0
--tserver_master_addrs=127.6.235.126:45407
--builtin_ntp_servers=127.6.235.84:42613
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260928 02:11:14.561354 11947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:11:14.554883 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.235.66:46937 (local address 127.6.235.126:45407)
0928 02:11:14.555059 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0928 02:11:14.555065 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 02:11:14.556718 (+ 1653us) server_negotiation.cc:407] Connection header received
0928 02:11:14.558873 (+ 2155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:11:14.558877 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:11:14.558926 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:11:14.559011 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:11:14.560481 (+ 1470us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:11:14.560486 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:11:14.560489 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 02:11:14.560503 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:11:14.560533 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 02:11:14.560544 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:11:14.560643 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0928 02:11:14.560943 (+ 300us) server_negotiation.cc:299] Negotiation successful
0928 02:11:14.561007 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260928 02:11:14.563297 12078 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:11:14.555486 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.235.126:45407 (local address 127.6.235.66:46937)
0928 02:11:14.556533 (+ 1047us) negotiation.cc:107] Waiting for socket to connect
0928 02:11:14.556581 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0928 02:11:14.558583 (+ 2002us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:11:14.559274 (+ 691us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:11:14.559282 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:11:14.559383 (+ 101us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:11:14.560121 (+ 738us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 02:11:14.560125 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:11:14.560177 (+ 52us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 02:11:14.560295 (+ 118us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:11:14.560650 (+ 355us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:11:14.560661 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:11:14.560671 (+ 10us) client_negotiation.cc:769] Sending connection context
0928 02:11:14.561710 (+ 1039us) client_negotiation.cc:240] Negotiation successful
0928 02:11:14.561989 (+ 279us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":435,"thread_start_us":254,"threads_started":1}
I20260928 02:11:14.564719 12076 heartbeater.cc:347] Connected to a master server at 127.6.235.126:45407
I20260928 02:11:14.565073 12076 heartbeater.cc:464] Registering TS with master...
I20260928 02:11:14.565781 12076 heartbeater.cc:511] Master 127.6.235.126:45407 requested a full tablet report, sending...
I20260928 02:11:14.567619 11759 ts_manager.cc:194] Registered new tserver with Master: ca4d5f9b932a477fa95363975fb8a511 (127.6.235.66:34425)
W20260928 02:11:14.909550 12079 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:11:14.909854 12079 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:11:14.909951 12079 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:11:14.917505 12079 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:11:14.917629 12079 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:11:14.917750 12079 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.235.67
I20260928 02:11:14.927454 12079 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.235.84:42613
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790561432251112-7085-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790561432251112-7085-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.235.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790561432251112-7085-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.6.235.67
--webserver_port=0
--tserver_master_addrs=127.6.235.126:45407
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.235.67
--log_dir=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790561432251112-7085-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 00:51:59 UTC on ccycloud.quasar-axgkwn.root.comops.site
build id 4676464
ASAN enabled
I20260928 02:11:14.928966 12079 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:11:14.931061 12079 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:11:14.946249 12084 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:11:14.946091 12087 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:11:14.946172 12085 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:11:14.950807 12079 server_base.cc:1034] running on GCE node
I20260928 02:11:14.951551 12079 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:11:14.952644 12079 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:11:14.953920 12079 hybrid_clock.cc:648] HybridClock initialized: now 1790561474953786 us; error 120 us; skew 500 ppm
I20260928 02:11:14.954332 12079 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260928 02:11:14.956815 12079 webserver.cc:460] Webserver started at http://127.6.235.67:39423/ using document root <none> and password file <none>
I20260928 02:11:14.957587 12079 fs_manager.cc:359] Metadata directory not provided
I20260928 02:11:14.957734 12079 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:11:14.958124 12079 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:11:14.961289 12079 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790561432251112-7085-0/minicluster-data/ts-2/data/instance:
uuid: "783b5fc6b38146508c8bd67c566c9d32"
format_stamp: "Formatted at 2026-09-28 02:11:14 on dist-test-slave-86gv"
server_key: "a98d819eaba9605b0b8e9a82f61a2d75"
server_key_iv: "01cf077bd0ecc5726e1163df3c966d60"
server_key_version: "encryptionkey@0"
I20260928 02:11:14.962133 12079 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790561432251112-7085-0/minicluster-data/ts-2/wal/instance:
uuid: "783b5fc6b38146508c8bd67c566c9d32"
format_stamp: "Formatted at 2026-09-28 02:11:14 on dist-test-slave-86gv"
server_key: "a98d819eaba9605b0b8e9a82f61a2d75"
server_key_iv: "01cf077bd0ecc5726e1163df3c966d60"
server_key_version: "encryptionkey@0"
I20260928 02:11:14.967749 12079 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260928 02:11:14.971731 12093 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:11:14.973520 12079 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 02:11:14.973729 12079 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790561432251112-7085-0/minicluster-data/ts-2/data,/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790561432251112-7085-0/minicluster-data/ts-2/wal
uuid: "783b5fc6b38146508c8bd67c566c9d32"
format_stamp: "Formatted at 2026-09-28 02:11:14 on dist-test-slave-86gv"
server_key: "a98d819eaba9605b0b8e9a82f61a2d75"
server_key_iv: "01cf077bd0ecc5726e1163df3c966d60"
server_key_version: "encryptionkey@0"
I20260928 02:11:14.973968 12079 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790561432251112-7085-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790561432251112-7085-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790561432251112-7085-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:11:14.980072 12079 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:11:14.981118 12079 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:11:14.981397 12079 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:11:14.982352 12079 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:11:14.983888 12079 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:11:14.984017 12079 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:11:14.984165 12079 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:11:14.984254 12079 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:11:15.018597 12079 rpc_server.cc:307] RPC server started. Bound to: 127.6.235.67:43957
I20260928 02:11:15.018648 12205 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.235.67:43957 every 8 connection(s)
I20260928 02:11:15.020011 12079 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790561432251112-7085-0/minicluster-data/ts-2/data/info.pb
I20260928 02:11:15.024670 7085 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTlaUuc/build/asan/bin/kudu as pid 12079
I20260928 02:11:15.024853 7085 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790561432251112-7085-0/minicluster-data/ts-2/wal/instance
I20260928 02:11:15.025315 7085 external_mini_cluster.cc:1428] Setting key 83a7abb481834a7121a4b0a8dc30075f
I20260928 02:11:15.029654 11947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:11:15.022694 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.235.67:58109 (local address 127.6.235.126:45407)
0928 02:11:15.022917 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0928 02:11:15.022922 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 02:11:15.024502 (+ 1580us) server_negotiation.cc:407] Connection header received
0928 02:11:15.026889 (+ 2387us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:11:15.026895 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:11:15.026980 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:11:15.027120 (+ 140us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:11:15.028780 (+ 1660us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:11:15.028787 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:11:15.028792 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 02:11:15.028813 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:11:15.028864 (+ 51us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 02:11:15.028877 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:11:15.029061 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0928 02:11:15.029271 (+ 210us) server_negotiation.cc:299] Negotiation successful
0928 02:11:15.029340 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":98}
I20260928 02:11:15.030721 12208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:11:15.023275 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.235.126:45407 (local address 127.6.235.67:58109)
0928 02:11:15.024312 (+ 1037us) negotiation.cc:107] Waiting for socket to connect
0928 02:11:15.024361 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0928 02:11:15.026506 (+ 2145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:11:15.027419 (+ 913us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:11:15.027434 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:11:15.027577 (+ 143us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:11:15.028422 (+ 845us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 02:11:15.028427 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:11:15.028485 (+ 58us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 02:11:15.028605 (+ 120us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:11:15.029021 (+ 416us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:11:15.029032 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:11:15.029043 (+ 11us) client_negotiation.cc:769] Sending connection context
0928 02:11:15.029367 (+ 324us) client_negotiation.cc:240] Negotiation successful
0928 02:11:15.029671 (+ 304us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":532,"thread_start_us":316,"threads_started":1}
I20260928 02:11:15.032245 12206 heartbeater.cc:347] Connected to a master server at 127.6.235.126:45407
I20260928 02:11:15.032563 12206 heartbeater.cc:464] Registering TS with master...
I20260928 02:11:15.033262 12206 heartbeater.cc:511] Master 127.6.235.126:45407 requested a full tablet report, sending...
I20260928 02:11:15.034752 11759 ts_manager.cc:194] Registered new tserver with Master: 783b5fc6b38146508c8bd67c566c9d32 (127.6.235.67:43957)
I20260928 02:11:15.041203 7085 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 02:11:15.048489 11947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:11:15.045893 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60970 (local address 127.6.235.126:45407)
0928 02:11:15.046063 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0928 02:11:15.046067 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:11:15.046603 (+ 536us) server_negotiation.cc:407] Connection header received
0928 02:11:15.046733 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:11:15.046738 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:11:15.046794 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:11:15.046884 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:11:15.047922 (+ 1038us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:11:15.047927 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:11:15.047931 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 02:11:15.047949 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:11:15.047990 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 02:11:15.048002 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:11:15.048190 (+ 188us) server_negotiation.cc:1035] Waiting for connection context
0928 02:11:15.048230 (+ 40us) server_negotiation.cc:299] Negotiation successful
0928 02:11:15.048277 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260928 02:11:15.052718 11759 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:60970:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 02:11:15.059942 11759 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 02:11:15.078943 12221 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:11:15.074090 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60690 (local address 127.6.235.66:34425)
0928 02:11:15.074661 (+ 571us) server_negotiation.cc:206] Beginning negotiation
0928 02:11:15.074671 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 02:11:15.075672 (+ 1001us) server_negotiation.cc:407] Connection header received
0928 02:11:15.075745 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:11:15.075754 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:11:15.075968 (+ 214us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:11:15.076137 (+ 169us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:11:15.077660 (+ 1523us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:11:15.077670 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:11:15.077685 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 02:11:15.077711 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:11:15.077771 (+ 60us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 02:11:15.077794 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:11:15.077922 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0928 02:11:15.078424 (+ 502us) server_negotiation.cc:299] Negotiation successful
0928 02:11:15.078537 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":401,"thread_start_us":789,"threads_started":1}
I20260928 02:11:15.078967 12218 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:11:15.073123 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.235.67:43957 (local address 127.0.0.1:40142)
0928 02:11:15.074480 (+ 1357us) negotiation.cc:107] Waiting for socket to connect
0928 02:11:15.074531 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0928 02:11:15.074755 (+ 224us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:11:15.076514 (+ 1759us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:11:15.076522 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:11:15.076556 (+ 34us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:11:15.077637 (+ 1081us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 02:11:15.077640 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:11:15.077654 (+ 14us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 02:11:15.077690 (+ 36us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:11:15.078237 (+ 547us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:11:15.078243 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:11:15.078250 (+ 7us) client_negotiation.cc:769] Sending connection context
0928 02:11:15.078402 (+ 152us) client_negotiation.cc:240] Negotiation successful
0928 02:11:15.078498 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1059,"thread_start_us":1223,"threads_started":1}
I20260928 02:11:15.079605 12220 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:11:15.073459 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.235.66:34425 (local address 127.0.0.1:60690)
0928 02:11:15.075453 (+ 1994us) negotiation.cc:107] Waiting for socket to connect
0928 02:11:15.075470 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0928 02:11:15.075558 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:11:15.076151 (+ 593us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:11:15.076162 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:11:15.076280 (+ 118us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:11:15.077279 (+ 999us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 02:11:15.077286 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:11:15.077321 (+ 35us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 02:11:15.077409 (+ 88us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:11:15.077896 (+ 487us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:11:15.077911 (+ 15us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:11:15.077924 (+ 13us) client_negotiation.cc:769] Sending connection context
0928 02:11:15.079170 (+ 1246us) client_negotiation.cc:240] Negotiation successful
0928 02:11:15.079223 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1856,"thread_start_us":1707,"threads_started":1}
I20260928 02:11:15.080061 12219 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:11:15.073311 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40142 (local address 127.6.235.67:43957)
0928 02:11:15.074079 (+ 768us) server_negotiation.cc:206] Beginning negotiation
0928 02:11:15.074089 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 02:11:15.074643 (+ 554us) server_negotiation.cc:407] Connection header received
0928 02:11:15.074913 (+ 270us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:11:15.074922 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:11:15.075206 (+ 284us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:11:15.075368 (+ 162us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:11:15.077869 (+ 2501us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:11:15.077877 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:11:15.077889 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 02:11:15.077907 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:11:15.077967 (+ 60us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 02:11:15.077991 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:11:15.078179 (+ 188us) server_negotiation.cc:1035] Waiting for connection context
0928 02:11:15.079578 (+ 1399us) server_negotiation.cc:299] Negotiation successful
0928 02:11:15.079705 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":570,"thread_start_us":406,"threads_started":1}
I20260928 02:11:15.082840 12222 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:11:15.075499 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.235.65:34135 (local address 127.0.0.1:34980)
0928 02:11:15.078390 (+ 2891us) negotiation.cc:107] Waiting for socket to connect
0928 02:11:15.078407 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0928 02:11:15.078499 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:11:15.080053 (+ 1554us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:11:15.080057 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:11:15.080079 (+ 22us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:11:15.081666 (+ 1587us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 02:11:15.081669 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:11:15.081686 (+ 17us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 02:11:15.081727 (+ 41us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:11:15.082246 (+ 519us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:11:15.082250 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:11:15.082256 (+ 6us) client_negotiation.cc:769] Sending connection context
0928 02:11:15.082378 (+ 122us) client_negotiation.cc:240] Negotiation successful
0928 02:11:15.082437 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2762,"thread_start_us":2447,"threads_started":1}
I20260928 02:11:15.083707 12223 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:11:15.076018 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34980 (local address 127.6.235.65:34135)
0928 02:11:15.077478 (+ 1460us) server_negotiation.cc:206] Beginning negotiation
0928 02:11:15.077491 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0928 02:11:15.078605 (+ 1114us) server_negotiation.cc:407] Connection header received
0928 02:11:15.078778 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:11:15.078786 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:11:15.078998 (+ 212us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:11:15.079149 (+ 151us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:11:15.081910 (+ 2761us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:11:15.081919 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:11:15.081932 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 02:11:15.081953 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:11:15.082036 (+ 83us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 02:11:15.082061 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:11:15.082195 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0928 02:11:15.083188 (+ 993us) server_negotiation.cc:299] Negotiation successful
0928 02:11:15.083302 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1261,"thread_start_us":1059,"threads_started":1}
I20260928 02:11:15.083923 12141 tablet_service.cc:1467] Processing CreateTablet for tablet 6f8053a9a26a4a168aee4ebe6343575a (DEFAULT_TABLE table=test-table [id=e7d90cbe73574f339298932c75d27249]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:11:15.083880 12011 tablet_service.cc:1467] Processing CreateTablet for tablet 6f8053a9a26a4a168aee4ebe6343575a (DEFAULT_TABLE table=test-table [id=e7d90cbe73574f339298932c75d27249]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:11:15.085390 12141 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6f8053a9a26a4a168aee4ebe6343575a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:11:15.085397 12011 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6f8053a9a26a4a168aee4ebe6343575a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:11:15.087570 11880 tablet_service.cc:1467] Processing CreateTablet for tablet 6f8053a9a26a4a168aee4ebe6343575a (DEFAULT_TABLE table=test-table [id=e7d90cbe73574f339298932c75d27249]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:11:15.088996 11880 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6f8053a9a26a4a168aee4ebe6343575a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:11:15.099427 12224 tablet_bootstrap.cc:492] T 6f8053a9a26a4a168aee4ebe6343575a P ca4d5f9b932a477fa95363975fb8a511: Bootstrap starting.
I20260928 02:11:15.101468 12225 tablet_bootstrap.cc:492] T 6f8053a9a26a4a168aee4ebe6343575a P 783b5fc6b38146508c8bd67c566c9d32: Bootstrap starting.
I20260928 02:11:15.102209 11945 heartbeater.cc:503] Master 127.6.235.126:45407 was elected leader, sending a full tablet report...
I20260928 02:11:15.104646 12224 tablet_bootstrap.cc:654] T 6f8053a9a26a4a168aee4ebe6343575a P ca4d5f9b932a477fa95363975fb8a511: Neither blocks nor log segments found. Creating new log.
I20260928 02:11:15.105050 12226 tablet_bootstrap.cc:492] T 6f8053a9a26a4a168aee4ebe6343575a P 0b983ff76977406fb78575b56e369e6c: Bootstrap starting.
I20260928 02:11:15.105332 12225 tablet_bootstrap.cc:654] T 6f8053a9a26a4a168aee4ebe6343575a P 783b5fc6b38146508c8bd67c566c9d32: Neither blocks nor log segments found. Creating new log.
I20260928 02:11:15.106454 12224 log.cc:824] T 6f8053a9a26a4a168aee4ebe6343575a P ca4d5f9b932a477fa95363975fb8a511: Log is configured to *not* fsync() on all Append() calls
I20260928 02:11:15.106472 12225 log.cc:824] T 6f8053a9a26a4a168aee4ebe6343575a P 783b5fc6b38146508c8bd67c566c9d32: Log is configured to *not* fsync() on all Append() calls
I20260928 02:11:15.108347 12226 tablet_bootstrap.cc:654] T 6f8053a9a26a4a168aee4ebe6343575a P 0b983ff76977406fb78575b56e369e6c: Neither blocks nor log segments found. Creating new log.
I20260928 02:11:15.109512 12226 log.cc:824] T 6f8053a9a26a4a168aee4ebe6343575a P 0b983ff76977406fb78575b56e369e6c: Log is configured to *not* fsync() on all Append() calls
I20260928 02:11:15.111901 12226 tablet_bootstrap.cc:492] T 6f8053a9a26a4a168aee4ebe6343575a P 0b983ff76977406fb78575b56e369e6c: No bootstrap required, opened a new log
I20260928 02:11:15.112144 12225 tablet_bootstrap.cc:492] T 6f8053a9a26a4a168aee4ebe6343575a P 783b5fc6b38146508c8bd67c566c9d32: No bootstrap required, opened a new log
I20260928 02:11:15.112164 12226 ts_tablet_manager.cc:1397] T 6f8053a9a26a4a168aee4ebe6343575a P 0b983ff76977406fb78575b56e369e6c: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260928 02:11:15.112483 12225 ts_tablet_manager.cc:1397] T 6f8053a9a26a4a168aee4ebe6343575a P 783b5fc6b38146508c8bd67c566c9d32: Time spent bootstrapping tablet: real 0.011s user 0.005s sys 0.002s
I20260928 02:11:15.115806 12224 tablet_bootstrap.cc:492] T 6f8053a9a26a4a168aee4ebe6343575a P ca4d5f9b932a477fa95363975fb8a511: No bootstrap required, opened a new log
I20260928 02:11:15.116163 12224 ts_tablet_manager.cc:1397] T 6f8053a9a26a4a168aee4ebe6343575a P ca4d5f9b932a477fa95363975fb8a511: Time spent bootstrapping tablet: real 0.017s user 0.003s sys 0.011s
I20260928 02:11:15.120021 12226 raft_consensus.cc:348] T 6f8053a9a26a4a168aee4ebe6343575a P 0b983ff76977406fb78575b56e369e6c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "783b5fc6b38146508c8bd67c566c9d32" member_type: VOTER last_known_addr { host: "127.6.235.67" port: 43957 } } peers { permanent_uuid: "ca4d5f9b932a477fa95363975fb8a511" member_type: VOTER last_known_addr { host: "127.6.235.66" port: 34425 } } peers { permanent_uuid: "0b983ff76977406fb78575b56e369e6c" member_type: VOTER last_known_addr { host: "127.6.235.65" port: 34135 } }
I20260928 02:11:15.120428 12226 raft_consensus.cc:374] T 6f8053a9a26a4a168aee4ebe6343575a P 0b983ff76977406fb78575b56e369e6c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:11:15.120549 12226 raft_consensus.cc:729] T 6f8053a9a26a4a168aee4ebe6343575a P 0b983ff76977406fb78575b56e369e6c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0b983ff76977406fb78575b56e369e6c, State: Initialized, Role: FOLLOWER
I20260928 02:11:15.121207 12226 consensus_queue.cc:260] T 6f8053a9a26a4a168aee4ebe6343575a P 0b983ff76977406fb78575b56e369e6c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "783b5fc6b38146508c8bd67c566c9d32" member_type: VOTER last_known_addr { host: "127.6.235.67" port: 43957 } } peers { permanent_uuid: "ca4d5f9b932a477fa95363975fb8a511" member_type: VOTER last_known_addr { host: "127.6.235.66" port: 34425 } } peers { permanent_uuid: "0b983ff76977406fb78575b56e369e6c" member_type: VOTER last_known_addr { host: "127.6.235.65" port: 34135 } }
I20260928 02:11:15.123262 12225 raft_consensus.cc:348] T 6f8053a9a26a4a168aee4ebe6343575a P 783b5fc6b38146508c8bd67c566c9d32 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "783b5fc6b38146508c8bd67c566c9d32" member_type: VOTER last_known_addr { host: "127.6.235.67" port: 43957 } } peers { permanent_uuid: "ca4d5f9b932a477fa95363975fb8a511" member_type: VOTER last_known_addr { host: "127.6.235.66" port: 34425 } } peers { permanent_uuid: "0b983ff76977406fb78575b56e369e6c" member_type: VOTER last_known_addr { host: "127.6.235.65" port: 34135 } }
I20260928 02:11:15.123715 12225 raft_consensus.cc:374] T 6f8053a9a26a4a168aee4ebe6343575a P 783b5fc6b38146508c8bd67c566c9d32 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:11:15.123817 12225 raft_consensus.cc:729] T 6f8053a9a26a4a168aee4ebe6343575a P 783b5fc6b38146508c8bd67c566c9d32 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 783b5fc6b38146508c8bd67c566c9d32, State: Initialized, Role: FOLLOWER
I20260928 02:11:15.124507 12225 consensus_queue.cc:260] T 6f8053a9a26a4a168aee4ebe6343575a P 783b5fc6b38146508c8bd67c566c9d32 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "783b5fc6b38146508c8bd67c566c9d32" member_type: VOTER last_known_addr { host: "127.6.235.67" port: 43957 } } peers { permanent_uuid: "ca4d5f9b932a477fa95363975fb8a511" member_type: VOTER last_known_addr { host: "127.6.235.66" port: 34425 } } peers { permanent_uuid: "0b983ff76977406fb78575b56e369e6c" member_type: VOTER last_known_addr { host: "127.6.235.65" port: 34135 } }
I20260928 02:11:15.125514 12226 ts_tablet_manager.cc:1428] T 6f8053a9a26a4a168aee4ebe6343575a P 0b983ff76977406fb78575b56e369e6c: Time spent starting tablet: real 0.013s user 0.008s sys 0.003s
I20260928 02:11:15.127260 12224 raft_consensus.cc:348] T 6f8053a9a26a4a168aee4ebe6343575a P ca4d5f9b932a477fa95363975fb8a511 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "783b5fc6b38146508c8bd67c566c9d32" member_type: VOTER last_known_addr { host: "127.6.235.67" port: 43957 } } peers { permanent_uuid: "ca4d5f9b932a477fa95363975fb8a511" member_type: VOTER last_known_addr { host: "127.6.235.66" port: 34425 } } peers { permanent_uuid: "0b983ff76977406fb78575b56e369e6c" member_type: VOTER last_known_addr { host: "127.6.235.65" port: 34135 } }
I20260928 02:11:15.127882 12224 raft_consensus.cc:374] T 6f8053a9a26a4a168aee4ebe6343575a P ca4d5f9b932a477fa95363975fb8a511 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:11:15.128010 12224 raft_consensus.cc:729] T 6f8053a9a26a4a168aee4ebe6343575a P ca4d5f9b932a477fa95363975fb8a511 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ca4d5f9b932a477fa95363975fb8a511, State: Initialized, Role: FOLLOWER
I20260928 02:11:15.128944 12224 consensus_queue.cc:260] T 6f8053a9a26a4a168aee4ebe6343575a P ca4d5f9b932a477fa95363975fb8a511 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "783b5fc6b38146508c8bd67c566c9d32" member_type: VOTER last_known_addr { host: "127.6.235.67" port: 43957 } } peers { permanent_uuid: "ca4d5f9b932a477fa95363975fb8a511" member_type: VOTER last_known_addr { host: "127.6.235.66" port: 34425 } } peers { permanent_uuid: "0b983ff76977406fb78575b56e369e6c" member_type: VOTER last_known_addr { host: "127.6.235.65" port: 34135 } }
I20260928 02:11:15.129629 12206 heartbeater.cc:503] Master 127.6.235.126:45407 was elected leader, sending a full tablet report...
I20260928 02:11:15.130039 12076 heartbeater.cc:503] Master 127.6.235.126:45407 was elected leader, sending a full tablet report...
I20260928 02:11:15.130791 12224 ts_tablet_manager.cc:1428] T 6f8053a9a26a4a168aee4ebe6343575a P ca4d5f9b932a477fa95363975fb8a511: Time spent starting tablet: real 0.014s user 0.010s sys 0.003s
I20260928 02:11:15.131690 12225 ts_tablet_manager.cc:1428] T 6f8053a9a26a4a168aee4ebe6343575a P 783b5fc6b38146508c8bd67c566c9d32: Time spent starting tablet: real 0.019s user 0.017s sys 0.002s
W20260928 02:11:15.271770 12207 tablet.cc:2369] T 6f8053a9a26a4a168aee4ebe6343575a P 783b5fc6b38146508c8bd67c566c9d32: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 02:11:15.304471 12077 tablet.cc:2369] T 6f8053a9a26a4a168aee4ebe6343575a P ca4d5f9b932a477fa95363975fb8a511: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 02:11:15.318476 12231 raft_consensus.cc:482] T 6f8053a9a26a4a168aee4ebe6343575a P 783b5fc6b38146508c8bd67c566c9d32 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 02:11:15.318729 12231 raft_consensus.cc:504] T 6f8053a9a26a4a168aee4ebe6343575a P 783b5fc6b38146508c8bd67c566c9d32 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "783b5fc6b38146508c8bd67c566c9d32" member_type: VOTER last_known_addr { host: "127.6.235.67" port: 43957 } } peers { permanent_uuid: "ca4d5f9b932a477fa95363975fb8a511" member_type: VOTER last_known_addr { host: "127.6.235.66" port: 34425 } } peers { permanent_uuid: "0b983ff76977406fb78575b56e369e6c" member_type: VOTER last_known_addr { host: "127.6.235.65" port: 34135 } }
I20260928 02:11:15.320348 12231 leader_election.cc:288] T 6f8053a9a26a4a168aee4ebe6343575a P 783b5fc6b38146508c8bd67c566c9d32 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ca4d5f9b932a477fa95363975fb8a511 (127.6.235.66:34425), 0b983ff76977406fb78575b56e369e6c (127.6.235.65:34135)
I20260928 02:11:15.323007 12208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:11:15.320686 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.235.66:34425 (local address 127.6.235.67:58163)
0928 02:11:15.320896 (+ 210us) negotiation.cc:107] Waiting for socket to connect
0928 02:11:15.320945 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0928 02:11:15.321100 (+ 155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:11:15.321426 (+ 326us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:11:15.321433 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:11:15.321455 (+ 22us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:11:15.322208 (+ 753us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 02:11:15.322213 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:11:15.322230 (+ 17us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 02:11:15.322279 (+ 49us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:11:15.322575 (+ 296us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:11:15.322579 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:11:15.322585 (+ 6us) client_negotiation.cc:769] Sending connection context
0928 02:11:15.322707 (+ 122us) client_negotiation.cc:240] Negotiation successful
0928 02:11:15.322762 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":82}
I20260928 02:11:15.323024 12221 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:11:15.320704 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.235.67:58163 (local address 127.6.235.66:34425)
0928 02:11:15.320919 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0928 02:11:15.320925 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 02:11:15.321039 (+ 114us) server_negotiation.cc:407] Connection header received
0928 02:11:15.321240 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:11:15.321245 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:11:15.321293 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:11:15.321393 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:11:15.322414 (+ 1021us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:11:15.322424 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:11:15.322427 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 02:11:15.322441 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:11:15.322469 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 02:11:15.322480 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:11:15.322596 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0928 02:11:15.322699 (+ 103us) server_negotiation.cc:299] Negotiation successful
0928 02:11:15.322756 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260928 02:11:15.324020 12223 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:11:15.321430 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.235.67:35677 (local address 127.6.235.65:34135)
0928 02:11:15.321667 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0928 02:11:15.321673 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 02:11:15.322083 (+ 410us) server_negotiation.cc:407] Connection header received
0928 02:11:15.322230 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:11:15.322233 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:11:15.322282 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:11:15.322403 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:11:15.323384 (+ 981us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:11:15.323391 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:11:15.323396 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 02:11:15.323414 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:11:15.323454 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 02:11:15.323468 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:11:15.323569 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0928 02:11:15.323705 (+ 136us) server_negotiation.cc:299] Negotiation successful
0928 02:11:15.323764 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
I20260928 02:11:15.324033 12233 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:11:15.321168 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.235.65:34135 (local address 127.6.235.67:35677)
0928 02:11:15.321953 (+ 785us) negotiation.cc:107] Waiting for socket to connect
0928 02:11:15.321966 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 02:11:15.322105 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:11:15.322420 (+ 315us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:11:15.322425 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:11:15.322449 (+ 24us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:11:15.323164 (+ 715us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 02:11:15.323169 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:11:15.323190 (+ 21us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 02:11:15.323237 (+ 47us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:11:15.323582 (+ 345us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:11:15.323589 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:11:15.323594 (+ 5us) client_negotiation.cc:769] Sending connection context
0928 02:11:15.323711 (+ 117us) client_negotiation.cc:240] Negotiation successful
0928 02:11:15.323748 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":696,"thread_start_us":470,"threads_started":1}
I20260928 02:11:15.324188 12031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6f8053a9a26a4a168aee4ebe6343575a" candidate_uuid: "783b5fc6b38146508c8bd67c566c9d32" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ca4d5f9b932a477fa95363975fb8a511" is_pre_election: true
I20260928 02:11:15.324792 12031 raft_consensus.cc:2454] T 6f8053a9a26a4a168aee4ebe6343575a P ca4d5f9b932a477fa95363975fb8a511 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 783b5fc6b38146508c8bd67c566c9d32 in term 0.
I20260928 02:11:15.325305 11900 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6f8053a9a26a4a168aee4ebe6343575a" candidate_uuid: "783b5fc6b38146508c8bd67c566c9d32" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0b983ff76977406fb78575b56e369e6c" is_pre_election: true
I20260928 02:11:15.325491 12094 leader_election.cc:302] T 6f8053a9a26a4a168aee4ebe6343575a P 783b5fc6b38146508c8bd67c566c9d32 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 783b5fc6b38146508c8bd67c566c9d32, ca4d5f9b932a477fa95363975fb8a511; no voters:
I20260928 02:11:15.325802 11900 raft_consensus.cc:2454] T 6f8053a9a26a4a168aee4ebe6343575a P 0b983ff76977406fb78575b56e369e6c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 783b5fc6b38146508c8bd67c566c9d32 in term 0.
I20260928 02:11:15.325873 12231 raft_consensus.cc:2781] T 6f8053a9a26a4a168aee4ebe6343575a P 783b5fc6b38146508c8bd67c566c9d32 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 02:11:15.326021 12231 raft_consensus.cc:482] T 6f8053a9a26a4a168aee4ebe6343575a P 783b5fc6b38146508c8bd67c566c9d32 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 02:11:15.326135 12231 raft_consensus.cc:3037] T 6f8053a9a26a4a168aee4ebe6343575a P 783b5fc6b38146508c8bd67c566c9d32 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:11:15.328316 12231 raft_consensus.cc:504] T 6f8053a9a26a4a168aee4ebe6343575a P 783b5fc6b38146508c8bd67c566c9d32 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "783b5fc6b38146508c8bd67c566c9d32" member_type: VOTER last_known_addr { host: "127.6.235.67" port: 43957 } } peers { permanent_uuid: "ca4d5f9b932a477fa95363975fb8a511" member_type: VOTER last_known_addr { host: "127.6.235.66" port: 34425 } } peers { permanent_uuid: "0b983ff76977406fb78575b56e369e6c" member_type: VOTER last_known_addr { host: "127.6.235.65" port: 34135 } }
I20260928 02:11:15.328881 12231 leader_election.cc:288] T 6f8053a9a26a4a168aee4ebe6343575a P 783b5fc6b38146508c8bd67c566c9d32 [CANDIDATE]: Term 1 election: Requested vote from peers ca4d5f9b932a477fa95363975fb8a511 (127.6.235.66:34425), 0b983ff76977406fb78575b56e369e6c (127.6.235.65:34135)
I20260928 02:11:15.329361 12031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6f8053a9a26a4a168aee4ebe6343575a" candidate_uuid: "783b5fc6b38146508c8bd67c566c9d32" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ca4d5f9b932a477fa95363975fb8a511"
I20260928 02:11:15.329412 11900 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6f8053a9a26a4a168aee4ebe6343575a" candidate_uuid: "783b5fc6b38146508c8bd67c566c9d32" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0b983ff76977406fb78575b56e369e6c"
I20260928 02:11:15.329653 12031 raft_consensus.cc:3037] T 6f8053a9a26a4a168aee4ebe6343575a P ca4d5f9b932a477fa95363975fb8a511 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:11:15.329668 11900 raft_consensus.cc:3037] T 6f8053a9a26a4a168aee4ebe6343575a P 0b983ff76977406fb78575b56e369e6c [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:11:15.331821 11900 raft_consensus.cc:2454] T 6f8053a9a26a4a168aee4ebe6343575a P 0b983ff76977406fb78575b56e369e6c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 783b5fc6b38146508c8bd67c566c9d32 in term 1.
I20260928 02:11:15.331833 12031 raft_consensus.cc:2454] T 6f8053a9a26a4a168aee4ebe6343575a P ca4d5f9b932a477fa95363975fb8a511 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 783b5fc6b38146508c8bd67c566c9d32 in term 1.
I20260928 02:11:15.332203 12094 leader_election.cc:302] T 6f8053a9a26a4a168aee4ebe6343575a P 783b5fc6b38146508c8bd67c566c9d32 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0b983ff76977406fb78575b56e369e6c, 783b5fc6b38146508c8bd67c566c9d32; no voters:
I20260928 02:11:15.332470 12231 raft_consensus.cc:2781] T 6f8053a9a26a4a168aee4ebe6343575a P 783b5fc6b38146508c8bd67c566c9d32 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:11:15.332945 12231 raft_consensus.cc:686] T 6f8053a9a26a4a168aee4ebe6343575a P 783b5fc6b38146508c8bd67c566c9d32 [term 1 LEADER]: Becoming Leader. State: Replica: 783b5fc6b38146508c8bd67c566c9d32, State: Running, Role: LEADER
I20260928 02:11:15.333411 12231 consensus_queue.cc:237] T 6f8053a9a26a4a168aee4ebe6343575a P 783b5fc6b38146508c8bd67c566c9d32 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "783b5fc6b38146508c8bd67c566c9d32" member_type: VOTER last_known_addr { host: "127.6.235.67" port: 43957 } } peers { permanent_uuid: "ca4d5f9b932a477fa95363975fb8a511" member_type: VOTER last_known_addr { host: "127.6.235.66" port: 34425 } } peers { permanent_uuid: "0b983ff76977406fb78575b56e369e6c" member_type: VOTER last_known_addr { host: "127.6.235.65" port: 34135 } }
I20260928 02:11:15.337853 11758 catalog_manager.cc:5495] T 6f8053a9a26a4a168aee4ebe6343575a P 783b5fc6b38146508c8bd67c566c9d32 reported cstate change: term changed from 0 to 1, leader changed from <none> to 783b5fc6b38146508c8bd67c566c9d32 (127.6.235.67). New cstate: current_term: 1 leader_uuid: "783b5fc6b38146508c8bd67c566c9d32" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "783b5fc6b38146508c8bd67c566c9d32" member_type: VOTER last_known_addr { host: "127.6.235.67" port: 43957 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ca4d5f9b932a477fa95363975fb8a511" member_type: VOTER last_known_addr { host: "127.6.235.66" port: 34425 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0b983ff76977406fb78575b56e369e6c" member_type: VOTER last_known_addr { host: "127.6.235.65" port: 34135 } health_report { overall_health: UNKNOWN } } }
W20260928 02:11:15.339363 11946 tablet.cc:2369] T 6f8053a9a26a4a168aee4ebe6343575a P 0b983ff76977406fb78575b56e369e6c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 02:11:15.381265 12219 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:11:15.379139 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40154 (local address 127.6.235.67:43957)
0928 02:11:15.379321 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0928 02:11:15.379326 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 02:11:15.379473 (+ 147us) server_negotiation.cc:407] Connection header received
0928 02:11:15.379651 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:11:15.379655 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:11:15.379737 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:11:15.379821 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:11:15.380642 (+ 821us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:11:15.380647 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:11:15.380650 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 02:11:15.380665 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:11:15.380696 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 02:11:15.380708 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:11:15.380808 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0928 02:11:15.381022 (+ 214us) server_negotiation.cc:299] Negotiation successful
0928 02:11:15.381071 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260928 02:11:15.389839 12031 raft_consensus.cc:1261] T 6f8053a9a26a4a168aee4ebe6343575a P ca4d5f9b932a477fa95363975fb8a511 [term 1 FOLLOWER]: Refusing update from remote peer 783b5fc6b38146508c8bd67c566c9d32: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 02:11:15.389870 11900 raft_consensus.cc:1261] T 6f8053a9a26a4a168aee4ebe6343575a P 0b983ff76977406fb78575b56e369e6c [term 1 FOLLOWER]: Refusing update from remote peer 783b5fc6b38146508c8bd67c566c9d32: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 02:11:15.390800 12234 consensus_queue.cc:1035] T 6f8053a9a26a4a168aee4ebe6343575a P 783b5fc6b38146508c8bd67c566c9d32 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0b983ff76977406fb78575b56e369e6c" member_type: VOTER last_known_addr { host: "127.6.235.65" port: 34135 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 02:11:15.391160 12231 consensus_queue.cc:1035] T 6f8053a9a26a4a168aee4ebe6343575a P 783b5fc6b38146508c8bd67c566c9d32 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ca4d5f9b932a477fa95363975fb8a511" member_type: VOTER last_known_addr { host: "127.6.235.66" port: 34425 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 02:11:15.407585 12238 mvcc.cc:205] Tried to move back new op lower bound from 7334139803186339840 to 7334139802968834048. Current Snapshot: MvccSnapshot[applied={T|T < 7334139803186339840}]
I20260928 02:11:15.419737 11759 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:60970:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 02:11:15.420083 11759 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:60970:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 02:11:15.423939 11759 catalog_manager.cc:5782] T 00000000000000000000000000000000 P a19969de5aa54329851490c5e575daa7: Sending DeleteTablet for 3 replicas of tablet 6f8053a9a26a4a168aee4ebe6343575a
I20260928 02:11:15.425432 12141 tablet_service.cc:1514] Processing DeleteTablet for tablet 6f8053a9a26a4a168aee4ebe6343575a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:11:15 UTC) from {username='slave'} at 127.0.0.1:40142
I20260928 02:11:15.425423 11880 tablet_service.cc:1514] Processing DeleteTablet for tablet 6f8053a9a26a4a168aee4ebe6343575a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:11:15 UTC) from {username='slave'} at 127.0.0.1:34980
I20260928 02:11:15.426375 7085 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskTlaUuc/build/asan/bin/kudu pid 11818
I20260928 02:11:15.426410 12011 tablet_service.cc:1514] Processing DeleteTablet for tablet 6f8053a9a26a4a168aee4ebe6343575a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:11:15 UTC) from {username='slave'} at 127.0.0.1:60690
I20260928 02:11:15.436094 12249 tablet_replica.cc:330] T 6f8053a9a26a4a168aee4ebe6343575a P ca4d5f9b932a477fa95363975fb8a511: stopping tablet replica
I20260928 02:11:15.437603 12249 raft_consensus.cc:2229] T 6f8053a9a26a4a168aee4ebe6343575a P ca4d5f9b932a477fa95363975fb8a511 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 02:11:15.439153 12249 raft_consensus.cc:2258] T 6f8053a9a26a4a168aee4ebe6343575a P ca4d5f9b932a477fa95363975fb8a511 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 02:11:15.443099 12249 ts_tablet_manager.cc:1905] T 6f8053a9a26a4a168aee4ebe6343575a P ca4d5f9b932a477fa95363975fb8a511: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 02:11:15.444731 12248 tablet_replica.cc:330] T 6f8053a9a26a4a168aee4ebe6343575a P 783b5fc6b38146508c8bd67c566c9d32: stopping tablet replica
I20260928 02:11:15.445351 12223 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:11:15.428054 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34988 (local address 127.6.235.65:34135)
0928 02:11:15.428279 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0928 02:11:15.428286 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 02:11:15.442553 (+ 14267us) server_negotiation.cc:407] Connection header received
0928 02:11:15.442751 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:11:15.442757 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:11:15.442881 (+ 124us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:11:15.443006 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:11:15.444245 (+ 1239us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:11:15.444251 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:11:15.444256 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 02:11:15.444278 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:11:15.444442 (+ 164us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 02:11:15.444471 (+ 29us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:11:15.444592 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0928 02:11:15.444875 (+ 283us) server_negotiation.cc:299] Negotiation successful
0928 02:11:15.444983 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260928 02:11:15.445678 12247 tablet_replica.cc:330] T 6f8053a9a26a4a168aee4ebe6343575a P 0b983ff76977406fb78575b56e369e6c: stopping tablet replica
I20260928 02:11:15.446507 11940 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 02:11:15.446593 12248 raft_consensus.cc:2229] T 6f8053a9a26a4a168aee4ebe6343575a P 783b5fc6b38146508c8bd67c566c9d32 [term 1 LEADER]: Raft consensus shutting down.
I20260928 02:11:15.447651 12248 raft_consensus.cc:2258] T 6f8053a9a26a4a168aee4ebe6343575a P 783b5fc6b38146508c8bd67c566c9d32 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 02:11:15.450203 12248 ts_tablet_manager.cc:1905] T 6f8053a9a26a4a168aee4ebe6343575a P 783b5fc6b38146508c8bd67c566c9d32: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 02:11:15.451493 12249 ts_tablet_manager.cc:1918] T 6f8053a9a26a4a168aee4ebe6343575a P ca4d5f9b932a477fa95363975fb8a511: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 02:11:15.451726 12249 log.cc:1196] T 6f8053a9a26a4a168aee4ebe6343575a P ca4d5f9b932a477fa95363975fb8a511: Deleting WAL directory at /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790561432251112-7085-0/minicluster-data/ts-1/wal/wals/6f8053a9a26a4a168aee4ebe6343575a
I20260928 02:11:15.452311 12249 ts_tablet_manager.cc:1939] T 6f8053a9a26a4a168aee4ebe6343575a P ca4d5f9b932a477fa95363975fb8a511: Deleting consensus metadata
I20260928 02:11:15.454432 11743 catalog_manager.cc:4841] TS ca4d5f9b932a477fa95363975fb8a511 (127.6.235.66:34425): tablet 6f8053a9a26a4a168aee4ebe6343575a (table test-table [id=e7d90cbe73574f339298932c75d27249]) successfully deleted
I20260928 02:11:15.457443 12248 ts_tablet_manager.cc:1918] T 6f8053a9a26a4a168aee4ebe6343575a P 783b5fc6b38146508c8bd67c566c9d32: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 02:11:15.458015 12248 log.cc:1196] T 6f8053a9a26a4a168aee4ebe6343575a P 783b5fc6b38146508c8bd67c566c9d32: Deleting WAL directory at /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790561432251112-7085-0/minicluster-data/ts-2/wal/wals/6f8053a9a26a4a168aee4ebe6343575a
I20260928 02:11:15.459184 12248 ts_tablet_manager.cc:1939] T 6f8053a9a26a4a168aee4ebe6343575a P 783b5fc6b38146508c8bd67c566c9d32: Deleting consensus metadata
I20260928 02:11:15.461155 11745 catalog_manager.cc:4841] TS 783b5fc6b38146508c8bd67c566c9d32 (127.6.235.67:43957): tablet 6f8053a9a26a4a168aee4ebe6343575a (table test-table [id=e7d90cbe73574f339298932c75d27249]) successfully deleted
I20260928 02:11:15.447031 12247 raft_consensus.cc:2229] T 6f8053a9a26a4a168aee4ebe6343575a P 0b983ff76977406fb78575b56e369e6c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 02:11:15.557585 12247 raft_consensus.cc:2258] T 6f8053a9a26a4a168aee4ebe6343575a P 0b983ff76977406fb78575b56e369e6c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 02:11:15.558444 7085 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTlaUuc/build/asan/bin/kudu with pid 11818
W20260928 02:11:15.578692 11743 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.6.235.65:34135 (error 108)
I20260928 02:11:15.579929 7085 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskTlaUuc/build/asan/bin/kudu pid 11949
I20260928 02:11:15.585062 12254 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:11:15.584091 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.235.65:34135 (local address 127.0.0.1:34994)
0928 02:11:15.584524 (+ 433us) negotiation.cc:107] Waiting for socket to connect
0928 02:11:15.584670 (+ 146us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.235.65:34135: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":322,"thread_start_us":245,"threads_started":1}
W20260928 02:11:15.585353 11743 catalog_manager.cc:4584] TS 0b983ff76977406fb78575b56e369e6c (127.6.235.65:34135): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 6f8053a9a26a4a168aee4ebe6343575a: Network error: Client connection negotiation failed: client connection to 127.6.235.65:34135: connect: Connection refused (error 111)
I20260928 02:11:15.586051 12221 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:11:15.582416 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60706 (local address 127.6.235.66:34425)
0928 02:11:15.582669 (+ 253us) server_negotiation.cc:206] Beginning negotiation
0928 02:11:15.582678 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 02:11:15.583790 (+ 1112us) server_negotiation.cc:407] Connection header received
0928 02:11:15.583895 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:11:15.583907 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:11:15.584015 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:11:15.584122 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:11:15.585299 (+ 1177us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:11:15.585306 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:11:15.585312 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 02:11:15.585332 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:11:15.585374 (+ 42us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 02:11:15.585390 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:11:15.585526 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0928 02:11:15.585682 (+ 156us) server_negotiation.cc:299] Negotiation successful
0928 02:11:15.585748 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260928 02:11:15.586778 12071 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 02:11:15.639454 12254 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:11:15.638861 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.235.65:34135 (local address 127.0.0.1:34996)
0928 02:11:15.639128 (+ 267us) negotiation.cc:107] Waiting for socket to connect
0928 02:11:15.639266 (+ 138us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.235.65:34135: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":103}
I20260928 02:11:15.640743 12254 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:11:15.640304 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.235.65:34135 (local address 127.0.0.1:35010)
0928 02:11:15.640478 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0928 02:11:15.640567 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.235.65:34135: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
I20260928 02:11:15.676430 7085 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTlaUuc/build/asan/bin/kudu with pid 11949
I20260928 02:11:15.699214 7085 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskTlaUuc/build/asan/bin/kudu pid 12079
I20260928 02:11:15.701125 12254 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:11:15.700638 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.235.65:34135 (local address 127.0.0.1:35020)
0928 02:11:15.700843 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0928 02:11:15.700952 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.235.65:34135: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
I20260928 02:11:15.702435 12254 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:11:15.701955 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.235.65:34135 (local address 127.0.0.1:35024)
0928 02:11:15.702171 (+ 216us) negotiation.cc:107] Waiting for socket to connect
0928 02:11:15.702256 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.235.65:34135: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":116}
I20260928 02:11:15.702471 12219 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:11:15.700262 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40168 (local address 127.6.235.67:43957)
0928 02:11:15.700453 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0928 02:11:15.700459 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 02:11:15.700482 (+ 23us) server_negotiation.cc:407] Connection header received
0928 02:11:15.700575 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:11:15.700592 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:11:15.700659 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:11:15.700787 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:11:15.701831 (+ 1044us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:11:15.701837 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:11:15.701842 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 02:11:15.701865 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:11:15.701893 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 02:11:15.701905 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:11:15.702052 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0928 02:11:15.702137 (+ 85us) server_negotiation.cc:299] Negotiation successful
0928 02:11:15.702194 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260928 02:11:15.703501 12201 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 02:11:15.783087 12254 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:11:15.782408 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.235.65:34135 (local address 127.0.0.1:35038)
0928 02:11:15.782778 (+ 370us) negotiation.cc:107] Waiting for socket to connect
0928 02:11:15.782906 (+ 128us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.235.65:34135: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":125}
I20260928 02:11:15.784387 12254 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:11:15.783860 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.235.65:34135 (local address 127.0.0.1:35050)
0928 02:11:15.784102 (+ 242us) negotiation.cc:107] Waiting for socket to connect
0928 02:11:15.784206 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.235.65:34135: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":125}
I20260928 02:11:15.821300 7085 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTlaUuc/build/asan/bin/kudu with pid 12079
I20260928 02:11:15.841960 7085 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskTlaUuc/build/asan/bin/kudu pid 11728
I20260928 02:11:15.843268 11789 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 02:11:15.948205 7085 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTlaUuc/build/asan/bin/kudu with pid 11728
2026-09-28T02:11:15Z chronyd exiting
[ OK ] SecurityITest.TestDisableAuthenticationEncryption (2918 ms)
[ RUN ] SecurityITest.TestJwtMiniCluster
I20260928 02:11:15.990351 7085 mini_oidc.cc:152] Starting JWKS server
I20260928 02:11:15.993022 7085 webserver.cc:460] Webserver started at https://127.0.0.1:36831/ using document root <none> and password file <none>
I20260928 02:11:15.993300 7085 mini_oidc.cc:189] Starting OIDC Discovery server
I20260928 02:11:15.993912 7085 webserver.cc:460] Webserver started at http://127.0.0.1:42229/ using document root <none> and password file <none>
2026-09-28T02:11:16Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T02:11:16Z Disabled control of system clock
I20260928 02:11:16.021852 7085 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTlaUuc/build/asan/bin/kudu
/tmp/dist-test-taskTlaUuc/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790561432251112-7085-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790561432251112-7085-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790561432251112-7085-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790561432251112-7085-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.235.126:39817
--webserver_interface=127.6.235.126
--webserver_port=0
--builtin_ntp_servers=127.6.235.84:38189
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.235.126:39817
--enable_jwt_token_auth=true
--jwks_url=https://localhost:36831/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790561432251112-7085-0/testchainca.cert with env {}
W20260928 02:11:16.350749 12263 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:11:16.351074 12263 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:11:16.351164 12263 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:11:16.358525 12263 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 02:11:16.358644 12263 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:11:16.358682 12263 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:11:16.358737 12263 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 02:11:16.358785 12263 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260928 02:11:16.358839 12263 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260928 02:11:16.358877 12263 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:36831/jwks.json
I20260928 02:11:16.368790 12263 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.235.84:38189
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790561432251112-7085-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790561432251112-7085-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.235.126:39817
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.235.126:39817
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:36831/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790561432251112-7085-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.235.126
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790561432251112-7085-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790561432251112-7085-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 00:51:59 UTC on ccycloud.quasar-axgkwn.root.comops.site
build id 4676464
ASAN enabled
I20260928 02:11:16.370112 12263 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:11:16.371526 12263 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:11:16.384073 12269 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:11:16.384138 12271 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:11:16.384028 12268 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:11:16.385193 12263 server_base.cc:1034] running on GCE node
I20260928 02:11:16.386008 12263 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:11:16.387478 12263 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:11:16.388680 12263 hybrid_clock.cc:648] HybridClock initialized: now 1790561476388622 us; error 61 us; skew 500 ppm
I20260928 02:11:16.389094 12263 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260928 02:11:16.409051 12263 webserver.cc:460] Webserver started at http://127.6.235.126:41461/ using document root <none> and password file <none>
I20260928 02:11:16.409855 12263 fs_manager.cc:359] Metadata directory not provided
I20260928 02:11:16.409982 12263 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:11:16.410302 12263 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:11:16.413775 12263 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790561432251112-7085-0/minicluster-data/master-0/data/instance:
uuid: "0c90861c5bc44293964cdd6435556d46"
format_stamp: "Formatted at 2026-09-28 02:11:16 on dist-test-slave-86gv"
server_key: "89e5843ba20617a14bb00c8476082d49"
server_key_iv: "2b1a51c0ab6d2925523eacf5b0c8d209"
server_key_version: "encryptionkey@0"
I20260928 02:11:16.414657 12263 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790561432251112-7085-0/minicluster-data/master-0/wal/instance:
uuid: "0c90861c5bc44293964cdd6435556d46"
format_stamp: "Formatted at 2026-09-28 02:11:16 on dist-test-slave-86gv"
server_key: "89e5843ba20617a14bb00c8476082d49"
server_key_iv: "2b1a51c0ab6d2925523eacf5b0c8d209"
server_key_version: "encryptionkey@0"
I20260928 02:11:16.420148 12263 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260928 02:11:16.425104 12277 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:11:16.426676 12263 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260928 02:11:16.426841 12263 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790561432251112-7085-0/minicluster-data/master-0/data,/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790561432251112-7085-0/minicluster-data/master-0/wal
uuid: "0c90861c5bc44293964cdd6435556d46"
format_stamp: "Formatted at 2026-09-28 02:11:16 on dist-test-slave-86gv"
server_key: "89e5843ba20617a14bb00c8476082d49"
server_key_iv: "2b1a51c0ab6d2925523eacf5b0c8d209"
server_key_version: "encryptionkey@0"
I20260928 02:11:16.427037 12263 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790561432251112-7085-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790561432251112-7085-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790561432251112-7085-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:11:16.461000 12263 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:11:16.461877 12263 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:11:16.462252 12263 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:11:16.482380 12263 rpc_server.cc:307] RPC server started. Bound to: 127.6.235.126:39817
I20260928 02:11:16.482484 12328 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.235.126:39817 every 8 connection(s)
I20260928 02:11:16.483899 12263 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790561432251112-7085-0/minicluster-data/master-0/data/info.pb
I20260928 02:11:16.487794 12329 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:11:16.493111 7085 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTlaUuc/build/asan/bin/kudu as pid 12263
I20260928 02:11:16.493312 7085 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790561432251112-7085-0/minicluster-data/master-0/wal/instance
I20260928 02:11:16.493927 7085 external_mini_cluster.cc:1428] Setting key a3cfae11882c3d8b619a26ae5c220763
I20260928 02:11:16.499053 12329 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0c90861c5bc44293964cdd6435556d46: Bootstrap starting.
I20260928 02:11:16.502380 12329 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0c90861c5bc44293964cdd6435556d46: Neither blocks nor log segments found. Creating new log.
I20260928 02:11:16.503489 12329 log.cc:824] T 00000000000000000000000000000000 P 0c90861c5bc44293964cdd6435556d46: Log is configured to *not* fsync() on all Append() calls
I20260928 02:11:16.506908 12329 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0c90861c5bc44293964cdd6435556d46: No bootstrap required, opened a new log
I20260928 02:11:16.507855 12332 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:11:16.495934 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50008 (local address 127.6.235.126:39817)
0928 02:11:16.497021 (+ 1087us) server_negotiation.cc:206] Beginning negotiation
0928 02:11:16.497041 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0928 02:11:16.497089 (+ 48us) server_negotiation.cc:407] Connection header received
0928 02:11:16.498733 (+ 1644us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:11:16.498757 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:11:16.499246 (+ 489us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:11:16.499666 (+ 420us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:11:16.500238 (+ 572us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:11:16.501636 (+ 1398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:11:16.503096 (+ 1460us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:11:16.503907 (+ 811us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:11:16.504123 (+ 216us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:11:16.505456 (+ 1333us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:11:16.505464 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:11:16.505481 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 02:11:16.506061 (+ 580us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:11:16.506143 (+ 82us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 02:11:16.506164 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:11:16.506264 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0928 02:11:16.506520 (+ 256us) server_negotiation.cc:299] Negotiation successful
0928 02:11:16.506892 (+ 372us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":567,"thread_start_us":328,"threads_started":1}
I20260928 02:11:16.512223 7085 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 02:11:16.515668 12329 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0c90861c5bc44293964cdd6435556d46 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0c90861c5bc44293964cdd6435556d46" member_type: VOTER last_known_addr { host: "127.6.235.126" port: 39817 } }
I20260928 02:11:16.516088 12329 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0c90861c5bc44293964cdd6435556d46 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:11:16.516207 12329 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0c90861c5bc44293964cdd6435556d46 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0c90861c5bc44293964cdd6435556d46, State: Initialized, Role: FOLLOWER
I20260928 02:11:16.516800 12329 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0c90861c5bc44293964cdd6435556d46 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0c90861c5bc44293964cdd6435556d46" member_type: VOTER last_known_addr { host: "127.6.235.126" port: 39817 } }
I20260928 02:11:16.517077 12329 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0c90861c5bc44293964cdd6435556d46 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 02:11:16.517226 12329 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0c90861c5bc44293964cdd6435556d46 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 02:11:16.517416 12329 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0c90861c5bc44293964cdd6435556d46 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:11:16.519533 12329 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0c90861c5bc44293964cdd6435556d46 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0c90861c5bc44293964cdd6435556d46" member_type: VOTER last_known_addr { host: "127.6.235.126" port: 39817 } }
I20260928 02:11:16.520092 12329 leader_election.cc:302] T 00000000000000000000000000000000 P 0c90861c5bc44293964cdd6435556d46 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0c90861c5bc44293964cdd6435556d46; no voters:
I20260928 02:11:16.520692 12329 leader_election.cc:288] T 00000000000000000000000000000000 P 0c90861c5bc44293964cdd6435556d46 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 02:11:16.520841 12334 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0c90861c5bc44293964cdd6435556d46 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:11:16.522315 12334 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0c90861c5bc44293964cdd6435556d46 [term 1 LEADER]: Becoming Leader. State: Replica: 0c90861c5bc44293964cdd6435556d46, State: Running, Role: LEADER
I20260928 02:11:16.522863 12334 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0c90861c5bc44293964cdd6435556d46 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0c90861c5bc44293964cdd6435556d46" member_type: VOTER last_known_addr { host: "127.6.235.126" port: 39817 } }
I20260928 02:11:16.523096 12329 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0c90861c5bc44293964cdd6435556d46 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 02:11:16.526039 12336 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0c90861c5bc44293964cdd6435556d46 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0c90861c5bc44293964cdd6435556d46. Latest consensus state: current_term: 1 leader_uuid: "0c90861c5bc44293964cdd6435556d46" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0c90861c5bc44293964cdd6435556d46" member_type: VOTER last_known_addr { host: "127.6.235.126" port: 39817 } } }
I20260928 02:11:16.526754 12336 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0c90861c5bc44293964cdd6435556d46 [sys.catalog]: This master's current role is: LEADER
I20260928 02:11:16.526396 12335 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0c90861c5bc44293964cdd6435556d46 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0c90861c5bc44293964cdd6435556d46" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0c90861c5bc44293964cdd6435556d46" member_type: VOTER last_known_addr { host: "127.6.235.126" port: 39817 } } }
I20260928 02:11:16.527511 12335 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0c90861c5bc44293964cdd6435556d46 [sys.catalog]: This master's current role is: LEADER
W20260928 02:11:16.540822 12350 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 0c90861c5bc44293964cdd6435556d46: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 02:11:16.541028 12350 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 02:11:16.541195 12342 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 02:11:16.543390 12342 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 02:11:16.552388 12342 catalog_manager.cc:1347] Generated new cluster ID: 4aa5b70dde9740e9ba60ae182fe16658
I20260928 02:11:16.552515 12342 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 02:11:16.567068 12342 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 02:11:16.568964 12342 catalog_manager.cc:1504] Loading token signing keys...
I20260928 02:11:16.583204 12342 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0c90861c5bc44293964cdd6435556d46: Generated new TSK 0
I20260928 02:11:16.584388 12342 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260928 02:11:16.630082 12359 webserver.cc:533] Webserver: sslize failed: error:14094415:SSL routines:ssl3_read_bytes:sslv3 alert certificate expired
E20260928 02:11:16.630259 12359 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260928 02:11:16.631546 12332 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:11:16.609149 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50014 (local address 127.6.235.126:39817)
0928 02:11:16.609372 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0928 02:11:16.609380 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 02:11:16.609821 (+ 441us) server_negotiation.cc:407] Connection header received
0928 02:11:16.610090 (+ 269us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:11:16.610096 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:11:16.610192 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:11:16.610359 (+ 167us) server_negotiation.cc:226] Negotiated authn=JWT
0928 02:11:16.610886 (+ 527us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:11:16.612046 (+ 1160us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:11:16.613580 (+ 1534us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:11:16.614316 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:11:16.614455 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:11:16.614956 (+ 501us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0928 02:11:16.630806 (+ 15850us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36831/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0928 02:11:16.631229 (+ 423us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:50014: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36831/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"server-negotiator.queue_time_us":72}
W20260928 02:11:16.631665 12332 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:50014: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36831/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
W20260928 02:11:16.633145 12358 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:11:16.609003 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.235.126:39817 (local address 127.0.0.1:50014)
0928 02:11:16.609708 (+ 705us) negotiation.cc:107] Waiting for socket to connect
0928 02:11:16.609724 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 02:11:16.609864 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:11:16.610408 (+ 544us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:11:16.610412 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:11:16.610422 (+ 10us) client_negotiation.cc:189] Negotiated authn=JWT
0928 02:11:16.610722 (+ 300us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:11:16.610730 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:11:16.612186 (+ 1456us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:11:16.612189 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:11:16.613419 (+ 1230us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:11:16.613431 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:11:16.614497 (+ 1066us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:11:16.614501 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:11:16.614622 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:11:16.614684 (+ 62us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0928 02:11:16.631133 (+ 16449us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36831/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0928 02:11:16.631450 (+ 317us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.6.235.126:39817: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36831/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"client-negotiator.queue_time_us":522,"thread_start_us":384,"threads_started":1}
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:597: Failure
Failed
Bad status: Runtime error: Could not connect to the cluster: Client connection negotiation failed: client connection to 127.6.235.126:39817: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36831/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Google Test trace:
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:593: Valid JWT
I20260928 02:11:16.635210 7085 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskTlaUuc/build/asan/bin/kudu pid 12263
I20260928 02:11:16.636394 12324 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 02:11:16.733477 7085 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTlaUuc/build/asan/bin/kudu with pid 12263
2026-09-28T02:11:16Z chronyd exiting
I20260928 02:11:16.785420 7085 test_util.cc:175] -----------------------------------------------
I20260928 02:11:16.785598 7085 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790561432251112-7085-0
[ FAILED ] SecurityITest.TestJwtMiniCluster (799 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithInvalidCert
I20260928 02:11:16.789906 7085 mini_oidc.cc:152] Starting JWKS server
I20260928 02:11:16.791988 7085 webserver.cc:460] Webserver started at https://127.0.0.1:34463/ using document root <none> and password file <none>
I20260928 02:11:16.792276 7085 mini_oidc.cc:189] Starting OIDC Discovery server
I20260928 02:11:16.793084 7085 webserver.cc:460] Webserver started at http://127.0.0.1:44799/ using document root <none> and password file <none>
2026-09-28T02:11:16Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T02:11:16Z Disabled control of system clock
I20260928 02:11:16.821522 7085 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTlaUuc/build/asan/bin/kudu
/tmp/dist-test-taskTlaUuc/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790561432251112-7085-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790561432251112-7085-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790561432251112-7085-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790561432251112-7085-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.235.126:39067
--webserver_interface=127.6.235.126
--webserver_port=0
--builtin_ntp_servers=127.6.235.84:44577
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.235.126:39067
--enable_jwt_token_auth=true
--jwks_url=https://localhost:34463/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790561432251112-7085-0/testchainca.cert with env {}
W20260928 02:11:17.153134 12366 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:11:17.153450 12366 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:11:17.153653 12366 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:11:17.161002 12366 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 02:11:17.161111 12366 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:11:17.161223 12366 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:11:17.161315 12366 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 02:11:17.161377 12366 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260928 02:11:17.161485 12366 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260928 02:11:17.161540 12366 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:34463/jwks.json
I20260928 02:11:17.171494 12366 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.235.84:44577
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790561432251112-7085-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790561432251112-7085-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.235.126:39067
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.235.126:39067
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:34463/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790561432251112-7085-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.235.126
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790561432251112-7085-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790561432251112-7085-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 00:51:59 UTC on ccycloud.quasar-axgkwn.root.comops.site
build id 4676464
ASAN enabled
I20260928 02:11:17.172994 12366 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:11:17.174482 12366 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:11:17.187005 12371 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:11:17.187019 12374 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:11:17.186997 12372 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:11:17.187798 12366 server_base.cc:1034] running on GCE node
I20260928 02:11:17.188707 12366 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:11:17.190248 12366 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:11:17.191447 12366 hybrid_clock.cc:648] HybridClock initialized: now 1790561477191407 us; error 35 us; skew 500 ppm
I20260928 02:11:17.191811 12366 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260928 02:11:17.194404 12366 webserver.cc:460] Webserver started at http://127.6.235.126:34061/ using document root <none> and password file <none>
I20260928 02:11:17.195186 12366 fs_manager.cc:359] Metadata directory not provided
I20260928 02:11:17.195282 12366 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:11:17.195657 12366 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:11:17.198748 12366 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790561432251112-7085-0/minicluster-data/master-0/data/instance:
uuid: "a84737aa44eb468c9fea7259710963c0"
format_stamp: "Formatted at 2026-09-28 02:11:17 on dist-test-slave-86gv"
server_key: "3c6770237c57953af2f7f7c5dc3eeaa5"
server_key_iv: "7760733c742718b417a827e9cd714ebb"
server_key_version: "encryptionkey@0"
I20260928 02:11:17.199584 12366 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790561432251112-7085-0/minicluster-data/master-0/wal/instance:
uuid: "a84737aa44eb468c9fea7259710963c0"
format_stamp: "Formatted at 2026-09-28 02:11:17 on dist-test-slave-86gv"
server_key: "3c6770237c57953af2f7f7c5dc3eeaa5"
server_key_iv: "7760733c742718b417a827e9cd714ebb"
server_key_version: "encryptionkey@0"
I20260928 02:11:17.205117 12366 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260928 02:11:17.208663 12380 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:11:17.210266 12366 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 02:11:17.210484 12366 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790561432251112-7085-0/minicluster-data/master-0/data,/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790561432251112-7085-0/minicluster-data/master-0/wal
uuid: "a84737aa44eb468c9fea7259710963c0"
format_stamp: "Formatted at 2026-09-28 02:11:17 on dist-test-slave-86gv"
server_key: "3c6770237c57953af2f7f7c5dc3eeaa5"
server_key_iv: "7760733c742718b417a827e9cd714ebb"
server_key_version: "encryptionkey@0"
I20260928 02:11:17.210706 12366 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790561432251112-7085-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790561432251112-7085-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790561432251112-7085-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:11:17.231446 12366 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:11:17.232369 12366 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:11:17.232712 12366 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:11:17.252156 12366 rpc_server.cc:307] RPC server started. Bound to: 127.6.235.126:39067
I20260928 02:11:17.252179 12431 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.235.126:39067 every 8 connection(s)
I20260928 02:11:17.253638 12366 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790561432251112-7085-0/minicluster-data/master-0/data/info.pb
I20260928 02:11:17.257689 12432 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:11:17.263059 7085 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTlaUuc/build/asan/bin/kudu as pid 12366
I20260928 02:11:17.263279 7085 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790561432251112-7085-0/minicluster-data/master-0/wal/instance
I20260928 02:11:17.263942 7085 external_mini_cluster.cc:1428] Setting key 164d5a09567dbf10d8ddddeff614c08f
I20260928 02:11:17.269390 12432 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a84737aa44eb468c9fea7259710963c0: Bootstrap starting.
I20260928 02:11:17.272650 12432 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a84737aa44eb468c9fea7259710963c0: Neither blocks nor log segments found. Creating new log.
I20260928 02:11:17.273790 12432 log.cc:824] T 00000000000000000000000000000000 P a84737aa44eb468c9fea7259710963c0: Log is configured to *not* fsync() on all Append() calls
I20260928 02:11:17.276832 12432 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a84737aa44eb468c9fea7259710963c0: No bootstrap required, opened a new log
I20260928 02:11:17.278165 12435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:11:17.266003 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58342 (local address 127.6.235.126:39067)
0928 02:11:17.267206 (+ 1203us) server_negotiation.cc:206] Beginning negotiation
0928 02:11:17.267218 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0928 02:11:17.267265 (+ 47us) server_negotiation.cc:407] Connection header received
0928 02:11:17.268971 (+ 1706us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:11:17.269008 (+ 37us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:11:17.269513 (+ 505us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:11:17.269981 (+ 468us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:11:17.270550 (+ 569us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:11:17.271891 (+ 1341us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:11:17.273311 (+ 1420us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:11:17.274118 (+ 807us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:11:17.274315 (+ 197us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:11:17.275705 (+ 1390us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:11:17.275712 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:11:17.275727 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 02:11:17.276294 (+ 567us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:11:17.276391 (+ 97us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 02:11:17.276413 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:11:17.276557 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0928 02:11:17.276840 (+ 283us) server_negotiation.cc:299] Negotiation successful
0928 02:11:17.277219 (+ 379us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":673,"thread_start_us":408,"threads_started":1}
I20260928 02:11:17.282433 7085 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 02:11:17.285614 12432 raft_consensus.cc:348] T 00000000000000000000000000000000 P a84737aa44eb468c9fea7259710963c0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a84737aa44eb468c9fea7259710963c0" member_type: VOTER last_known_addr { host: "127.6.235.126" port: 39067 } }
I20260928 02:11:17.286063 12432 raft_consensus.cc:374] T 00000000000000000000000000000000 P a84737aa44eb468c9fea7259710963c0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:11:17.286175 12432 raft_consensus.cc:729] T 00000000000000000000000000000000 P a84737aa44eb468c9fea7259710963c0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a84737aa44eb468c9fea7259710963c0, State: Initialized, Role: FOLLOWER
I20260928 02:11:17.286744 12432 consensus_queue.cc:260] T 00000000000000000000000000000000 P a84737aa44eb468c9fea7259710963c0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a84737aa44eb468c9fea7259710963c0" member_type: VOTER last_known_addr { host: "127.6.235.126" port: 39067 } }
I20260928 02:11:17.286984 12432 raft_consensus.cc:388] T 00000000000000000000000000000000 P a84737aa44eb468c9fea7259710963c0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 02:11:17.287128 12432 raft_consensus.cc:482] T 00000000000000000000000000000000 P a84737aa44eb468c9fea7259710963c0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 02:11:17.287297 12432 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a84737aa44eb468c9fea7259710963c0 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:11:17.289805 12432 raft_consensus.cc:504] T 00000000000000000000000000000000 P a84737aa44eb468c9fea7259710963c0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a84737aa44eb468c9fea7259710963c0" member_type: VOTER last_known_addr { host: "127.6.235.126" port: 39067 } }
I20260928 02:11:17.290323 12432 leader_election.cc:302] T 00000000000000000000000000000000 P a84737aa44eb468c9fea7259710963c0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a84737aa44eb468c9fea7259710963c0; no voters:
I20260928 02:11:17.290937 12432 leader_election.cc:288] T 00000000000000000000000000000000 P a84737aa44eb468c9fea7259710963c0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 02:11:17.291085 12437 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a84737aa44eb468c9fea7259710963c0 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:11:17.291779 12437 raft_consensus.cc:686] T 00000000000000000000000000000000 P a84737aa44eb468c9fea7259710963c0 [term 1 LEADER]: Becoming Leader. State: Replica: a84737aa44eb468c9fea7259710963c0, State: Running, Role: LEADER
I20260928 02:11:17.292367 12437 consensus_queue.cc:237] T 00000000000000000000000000000000 P a84737aa44eb468c9fea7259710963c0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a84737aa44eb468c9fea7259710963c0" member_type: VOTER last_known_addr { host: "127.6.235.126" port: 39067 } }
I20260928 02:11:17.293177 12432 sys_catalog.cc:564] T 00000000000000000000000000000000 P a84737aa44eb468c9fea7259710963c0 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 02:11:17.295373 12439 sys_catalog.cc:455] T 00000000000000000000000000000000 P a84737aa44eb468c9fea7259710963c0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a84737aa44eb468c9fea7259710963c0. Latest consensus state: current_term: 1 leader_uuid: "a84737aa44eb468c9fea7259710963c0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a84737aa44eb468c9fea7259710963c0" member_type: VOTER last_known_addr { host: "127.6.235.126" port: 39067 } } }
I20260928 02:11:17.295447 12438 sys_catalog.cc:455] T 00000000000000000000000000000000 P a84737aa44eb468c9fea7259710963c0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a84737aa44eb468c9fea7259710963c0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a84737aa44eb468c9fea7259710963c0" member_type: VOTER last_known_addr { host: "127.6.235.126" port: 39067 } } }
I20260928 02:11:17.295893 12438 sys_catalog.cc:458] T 00000000000000000000000000000000 P a84737aa44eb468c9fea7259710963c0 [sys.catalog]: This master's current role is: LEADER
I20260928 02:11:17.295898 12439 sys_catalog.cc:458] T 00000000000000000000000000000000 P a84737aa44eb468c9fea7259710963c0 [sys.catalog]: This master's current role is: LEADER
I20260928 02:11:17.296870 12445 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 02:11:17.302999 12445 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 02:11:17.312361 12445 catalog_manager.cc:1347] Generated new cluster ID: a11094afadff47a59dac881437710383
I20260928 02:11:17.312484 12445 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 02:11:17.322224 12445 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 02:11:17.323438 12445 catalog_manager.cc:1504] Loading token signing keys...
I20260928 02:11:17.334477 12445 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a84737aa44eb468c9fea7259710963c0: Generated new TSK 0
I20260928 02:11:17.335619 12445 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260928 02:11:17.370472 12462 webserver.cc:533] Webserver: sslize failed: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca
E20260928 02:11:17.370639 12462 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260928 02:11:17.371687 12435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:11:17.351703 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58352 (local address 127.6.235.126:39067)
0928 02:11:17.351937 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0928 02:11:17.351947 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 02:11:17.352452 (+ 505us) server_negotiation.cc:407] Connection header received
0928 02:11:17.352671 (+ 219us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:11:17.352676 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:11:17.352753 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:11:17.352963 (+ 210us) server_negotiation.cc:226] Negotiated authn=JWT
0928 02:11:17.353493 (+ 530us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:11:17.354205 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:11:17.356196 (+ 1991us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:11:17.356634 (+ 438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:11:17.356732 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:11:17.357343 (+ 611us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0928 02:11:17.371139 (+ 13796us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:34463/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0928 02:11:17.371492 (+ 353us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:58352: Failed to load JWKS: Error downloading JWKS from 'https://localhost:34463/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"server-negotiator.queue_time_us":72}
W20260928 02:11:17.371847 12435 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:58352: Failed to load JWKS: Error downloading JWKS from 'https://localhost:34463/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
W20260928 02:11:17.372174 12461 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:11:17.351681 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.235.126:39067 (local address 127.0.0.1:58352)
0928 02:11:17.352334 (+ 653us) negotiation.cc:107] Waiting for socket to connect
0928 02:11:17.352360 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0928 02:11:17.352511 (+ 151us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:11:17.352988 (+ 477us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:11:17.352994 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:11:17.353007 (+ 13us) client_negotiation.cc:189] Negotiated authn=JWT
0928 02:11:17.353334 (+ 327us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:11:17.353347 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:11:17.354347 (+ 1000us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:11:17.354353 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:11:17.356052 (+ 1699us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:11:17.356067 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:11:17.356802 (+ 735us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:11:17.356808 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:11:17.357034 (+ 226us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:11:17.357080 (+ 46us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0928 02:11:17.371408 (+ 14328us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:34463/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0928 02:11:17.371720 (+ 312us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.6.235.126:39067: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:34463/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"client-negotiator.queue_time_us":507,"thread_start_us":343,"threads_started":1}
I20260928 02:11:17.373297 7085 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskTlaUuc/build/asan/bin/kudu pid 12366
I20260928 02:11:17.374603 12427 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 02:11:17.471146 7085 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTlaUuc/build/asan/bin/kudu with pid 12366
2026-09-28T02:11:17Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithInvalidCert (738 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithUntrustedCert
I20260928 02:11:17.528404 7085 mini_oidc.cc:152] Starting JWKS server
I20260928 02:11:17.530520 7085 webserver.cc:460] Webserver started at https://127.0.0.1:36631/ using document root <none> and password file <none>
I20260928 02:11:17.530798 7085 mini_oidc.cc:189] Starting OIDC Discovery server
I20260928 02:11:17.531517 7085 webserver.cc:460] Webserver started at http://127.0.0.1:40727/ using document root <none> and password file <none>
2026-09-28T02:11:17Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T02:11:17Z Disabled control of system clock
I20260928 02:11:17.562881 7085 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTlaUuc/build/asan/bin/kudu
/tmp/dist-test-taskTlaUuc/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790561432251112-7085-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790561432251112-7085-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790561432251112-7085-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790561432251112-7085-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.235.126:46541
--webserver_interface=127.6.235.126
--webserver_port=0
--builtin_ntp_servers=127.6.235.84:45229
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.235.126:46541
--enable_jwt_token_auth=true
--jwks_url=https://localhost:36631/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260928 02:11:17.901407 12469 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:11:17.901801 12469 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:11:17.901973 12469 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:11:17.909415 12469 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 02:11:17.909574 12469 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:11:17.909665 12469 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:11:17.909776 12469 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 02:11:17.909863 12469 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260928 02:11:17.909952 12469 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260928 02:11:17.910033 12469 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:36631/jwks.json
I20260928 02:11:17.920655 12469 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.235.84:45229
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790561432251112-7085-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790561432251112-7085-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.235.126:46541
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.235.126:46541
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:36631/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790561432251112-7085-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.235.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790561432251112-7085-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 00:51:59 UTC on ccycloud.quasar-axgkwn.root.comops.site
build id 4676464
ASAN enabled
I20260928 02:11:17.922113 12469 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:11:17.924120 12469 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:11:17.936340 12477 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:11:17.936537 12474 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:11:17.936461 12475 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:11:17.937759 12469 server_base.cc:1034] running on GCE node
I20260928 02:11:17.938638 12469 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:11:17.940114 12469 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:11:17.941311 12469 hybrid_clock.cc:648] HybridClock initialized: now 1790561477941265 us; error 37 us; skew 500 ppm
I20260928 02:11:17.941689 12469 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260928 02:11:17.944342 12469 webserver.cc:460] Webserver started at http://127.6.235.126:37237/ using document root <none> and password file <none>
I20260928 02:11:17.945245 12469 fs_manager.cc:359] Metadata directory not provided
I20260928 02:11:17.945379 12469 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:11:17.945750 12469 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:11:17.948715 12469 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790561432251112-7085-0/minicluster-data/master-0/data/instance:
uuid: "48f1642d08b244368650f26ccda300c2"
format_stamp: "Formatted at 2026-09-28 02:11:17 on dist-test-slave-86gv"
server_key: "94f51e1195211c427593a549e871fc8e"
server_key_iv: "20fdaadf7f7f28285389566e66f09a5e"
server_key_version: "encryptionkey@0"
I20260928 02:11:17.949635 12469 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790561432251112-7085-0/minicluster-data/master-0/wal/instance:
uuid: "48f1642d08b244368650f26ccda300c2"
format_stamp: "Formatted at 2026-09-28 02:11:17 on dist-test-slave-86gv"
server_key: "94f51e1195211c427593a549e871fc8e"
server_key_iv: "20fdaadf7f7f28285389566e66f09a5e"
server_key_version: "encryptionkey@0"
I20260928 02:11:17.954841 12469 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260928 02:11:17.958245 12483 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:11:17.959573 12469 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260928 02:11:17.959744 12469 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790561432251112-7085-0/minicluster-data/master-0/data,/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790561432251112-7085-0/minicluster-data/master-0/wal
uuid: "48f1642d08b244368650f26ccda300c2"
format_stamp: "Formatted at 2026-09-28 02:11:17 on dist-test-slave-86gv"
server_key: "94f51e1195211c427593a549e871fc8e"
server_key_iv: "20fdaadf7f7f28285389566e66f09a5e"
server_key_version: "encryptionkey@0"
I20260928 02:11:17.959969 12469 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790561432251112-7085-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790561432251112-7085-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790561432251112-7085-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:11:17.976686 12469 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:11:17.977571 12469 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:11:17.977886 12469 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:11:17.995994 12469 rpc_server.cc:307] RPC server started. Bound to: 127.6.235.126:46541
I20260928 02:11:17.996016 12534 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.235.126:46541 every 8 connection(s)
I20260928 02:11:17.997452 12469 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790561432251112-7085-0/minicluster-data/master-0/data/info.pb
I20260928 02:11:18.001307 12535 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:11:18.003597 7085 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTlaUuc/build/asan/bin/kudu as pid 12469
I20260928 02:11:18.003808 7085 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790561432251112-7085-0/minicluster-data/master-0/wal/instance
I20260928 02:11:18.004592 7085 external_mini_cluster.cc:1428] Setting key bedf343bbf0b36685fb98f63c25bd6a4
I20260928 02:11:18.013907 12535 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 48f1642d08b244368650f26ccda300c2: Bootstrap starting.
I20260928 02:11:18.017128 12535 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 48f1642d08b244368650f26ccda300c2: Neither blocks nor log segments found. Creating new log.
I20260928 02:11:18.018298 12535 log.cc:824] T 00000000000000000000000000000000 P 48f1642d08b244368650f26ccda300c2: Log is configured to *not* fsync() on all Append() calls
I20260928 02:11:18.019084 12538 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:11:18.006836 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38924 (local address 127.6.235.126:46541)
0928 02:11:18.007924 (+ 1088us) server_negotiation.cc:206] Beginning negotiation
0928 02:11:18.007937 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0928 02:11:18.007990 (+ 53us) server_negotiation.cc:407] Connection header received
0928 02:11:18.009750 (+ 1760us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:11:18.009775 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:11:18.010305 (+ 530us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:11:18.010768 (+ 463us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:11:18.011393 (+ 625us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:11:18.012706 (+ 1313us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:11:18.014136 (+ 1430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:11:18.014896 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:11:18.015099 (+ 203us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:11:18.016395 (+ 1296us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:11:18.016403 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:11:18.016417 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 02:11:18.017018 (+ 601us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:11:18.017122 (+ 104us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 02:11:18.017143 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:11:18.017318 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0928 02:11:18.017581 (+ 263us) server_negotiation.cc:299] Negotiation successful
0928 02:11:18.017892 (+ 311us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":604,"thread_start_us":356,"threads_started":1}
I20260928 02:11:18.021723 12535 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 48f1642d08b244368650f26ccda300c2: No bootstrap required, opened a new log
I20260928 02:11:18.023319 7085 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 02:11:18.029626 12535 raft_consensus.cc:348] T 00000000000000000000000000000000 P 48f1642d08b244368650f26ccda300c2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "48f1642d08b244368650f26ccda300c2" member_type: VOTER last_known_addr { host: "127.6.235.126" port: 46541 } }
I20260928 02:11:18.030026 12535 raft_consensus.cc:374] T 00000000000000000000000000000000 P 48f1642d08b244368650f26ccda300c2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:11:18.030136 12535 raft_consensus.cc:729] T 00000000000000000000000000000000 P 48f1642d08b244368650f26ccda300c2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 48f1642d08b244368650f26ccda300c2, State: Initialized, Role: FOLLOWER
I20260928 02:11:18.030776 12535 consensus_queue.cc:260] T 00000000000000000000000000000000 P 48f1642d08b244368650f26ccda300c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "48f1642d08b244368650f26ccda300c2" member_type: VOTER last_known_addr { host: "127.6.235.126" port: 46541 } }
I20260928 02:11:18.031060 12535 raft_consensus.cc:388] T 00000000000000000000000000000000 P 48f1642d08b244368650f26ccda300c2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 02:11:18.031219 12535 raft_consensus.cc:482] T 00000000000000000000000000000000 P 48f1642d08b244368650f26ccda300c2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 02:11:18.031394 12535 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 48f1642d08b244368650f26ccda300c2 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:11:18.033648 12535 raft_consensus.cc:504] T 00000000000000000000000000000000 P 48f1642d08b244368650f26ccda300c2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "48f1642d08b244368650f26ccda300c2" member_type: VOTER last_known_addr { host: "127.6.235.126" port: 46541 } }
I20260928 02:11:18.034217 12535 leader_election.cc:302] T 00000000000000000000000000000000 P 48f1642d08b244368650f26ccda300c2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 48f1642d08b244368650f26ccda300c2; no voters:
I20260928 02:11:18.034868 12535 leader_election.cc:288] T 00000000000000000000000000000000 P 48f1642d08b244368650f26ccda300c2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 02:11:18.035007 12540 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 48f1642d08b244368650f26ccda300c2 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:11:18.035979 12540 raft_consensus.cc:686] T 00000000000000000000000000000000 P 48f1642d08b244368650f26ccda300c2 [term 1 LEADER]: Becoming Leader. State: Replica: 48f1642d08b244368650f26ccda300c2, State: Running, Role: LEADER
I20260928 02:11:18.036492 12540 consensus_queue.cc:237] T 00000000000000000000000000000000 P 48f1642d08b244368650f26ccda300c2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "48f1642d08b244368650f26ccda300c2" member_type: VOTER last_known_addr { host: "127.6.235.126" port: 46541 } }
I20260928 02:11:18.037156 12535 sys_catalog.cc:564] T 00000000000000000000000000000000 P 48f1642d08b244368650f26ccda300c2 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 02:11:18.039507 12541 sys_catalog.cc:455] T 00000000000000000000000000000000 P 48f1642d08b244368650f26ccda300c2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "48f1642d08b244368650f26ccda300c2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "48f1642d08b244368650f26ccda300c2" member_type: VOTER last_known_addr { host: "127.6.235.126" port: 46541 } } }
I20260928 02:11:18.039533 12542 sys_catalog.cc:455] T 00000000000000000000000000000000 P 48f1642d08b244368650f26ccda300c2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 48f1642d08b244368650f26ccda300c2. Latest consensus state: current_term: 1 leader_uuid: "48f1642d08b244368650f26ccda300c2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "48f1642d08b244368650f26ccda300c2" member_type: VOTER last_known_addr { host: "127.6.235.126" port: 46541 } } }
I20260928 02:11:18.039965 12542 sys_catalog.cc:458] T 00000000000000000000000000000000 P 48f1642d08b244368650f26ccda300c2 [sys.catalog]: This master's current role is: LEADER
I20260928 02:11:18.039965 12541 sys_catalog.cc:458] T 00000000000000000000000000000000 P 48f1642d08b244368650f26ccda300c2 [sys.catalog]: This master's current role is: LEADER
I20260928 02:11:18.044095 12548 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 02:11:18.048949 12548 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 02:11:18.058714 12548 catalog_manager.cc:1347] Generated new cluster ID: 64e2f8d2f129413a9f60ecaad6d3585f
I20260928 02:11:18.058893 12548 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 02:11:18.090732 12548 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 02:11:18.092191 12548 catalog_manager.cc:1504] Loading token signing keys...
I20260928 02:11:18.099872 12548 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 48f1642d08b244368650f26ccda300c2: Generated new TSK 0
I20260928 02:11:18.100879 12548 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 02:11:18.130534 12538 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:11:18.122893 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38930 (local address 127.6.235.126:46541)
0928 02:11:18.123102 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0928 02:11:18.123108 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 02:11:18.123401 (+ 293us) server_negotiation.cc:407] Connection header received
0928 02:11:18.123675 (+ 274us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:11:18.123679 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:11:18.123748 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:11:18.123979 (+ 231us) server_negotiation.cc:226] Negotiated authn=JWT
0928 02:11:18.124477 (+ 498us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:11:18.125273 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:11:18.126587 (+ 1314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:11:18.127036 (+ 449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:11:18.127204 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:11:18.127753 (+ 549us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0928 02:11:18.129859 (+ 2106us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36631/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0928 02:11:18.130217 (+ 358us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:38930: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36631/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"server-negotiator.queue_time_us":55}
W20260928 02:11:18.130913 12538 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:38930: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36631/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
W20260928 02:11:18.133841 12564 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:11:18.122735 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.235.126:46541 (local address 127.0.0.1:38930)
0928 02:11:18.123272 (+ 537us) negotiation.cc:107] Waiting for socket to connect
0928 02:11:18.123292 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0928 02:11:18.123484 (+ 192us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:11:18.123988 (+ 504us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:11:18.123993 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:11:18.124005 (+ 12us) client_negotiation.cc:189] Negotiated authn=JWT
0928 02:11:18.124325 (+ 320us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:11:18.124333 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:11:18.125419 (+ 1086us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:11:18.125425 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:11:18.126442 (+ 1017us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:11:18.126452 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:11:18.127281 (+ 829us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:11:18.127291 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:11:18.127454 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:11:18.127487 (+ 33us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0928 02:11:18.130591 (+ 3104us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36631/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0928 02:11:18.133495 (+ 2904us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.6.235.126:46541: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36631/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"client-negotiator.queue_time_us":395,"mutex_wait_us":2596,"spinlock_wait_cycles":6528,"thread_start_us":381,"threads_started":1}
I20260928 02:11:18.134917 7085 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskTlaUuc/build/asan/bin/kudu pid 12469
I20260928 02:11:18.136516 12530 generic_service.cc:196] Checking for leaks (request via RPC)
E20260928 02:11:18.144788 12565 webserver.cc:533] Webserver: sslize failed:
E20260928 02:11:18.144968 12565 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260928 02:11:18.232288 7085 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTlaUuc/build/asan/bin/kudu with pid 12469
2026-09-28T02:11:18Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithUntrustedCert (760 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS
2026-09-28T02:11:18Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T02:11:18Z Disabled control of system clock
I20260928 02:11:18.319218 7085 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTlaUuc/build/asan/bin/kudu
/tmp/dist-test-taskTlaUuc/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790561432251112-7085-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790561432251112-7085-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790561432251112-7085-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790561432251112-7085-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.235.126:45767
--webserver_interface=127.6.235.126
--webserver_port=0
--builtin_ntp_servers=127.6.235.84:44595
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.235.126:45767
--enable_jwt_token_auth=true
--jwks_url=default.url
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790561432251112-7085-0/testchainca.cert with env {}
W20260928 02:11:18.663008 12570 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:11:18.663340 12570 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:11:18.663432 12570 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:11:18.670979 12570 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 02:11:18.671118 12570 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:11:18.671156 12570 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:11:18.671205 12570 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 02:11:18.671242 12570 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260928 02:11:18.671296 12570 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260928 02:11:18.671344 12570 flags.cc:425] Enabled experimental flag: --jwks_url=default.url
I20260928 02:11:18.681871 12570 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.235.84:44595
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790561432251112-7085-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790561432251112-7085-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.235.126:45767
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.235.126:45767
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=default.url
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790561432251112-7085-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.235.126
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790561432251112-7085-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790561432251112-7085-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 00:51:59 UTC on ccycloud.quasar-axgkwn.root.comops.site
build id 4676464
ASAN enabled
I20260928 02:11:18.683295 12570 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:11:18.684616 12570 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:11:18.697397 12575 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:11:18.697365 12578 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:11:18.697365 12576 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:11:18.698300 12570 server_base.cc:1034] running on GCE node
I20260928 02:11:18.699134 12570 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:11:18.700542 12570 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:11:18.701768 12570 hybrid_clock.cc:648] HybridClock initialized: now 1790561478701675 us; error 79 us; skew 500 ppm
I20260928 02:11:18.702240 12570 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260928 02:11:18.704854 12570 webserver.cc:460] Webserver started at http://127.6.235.126:45291/ using document root <none> and password file <none>
I20260928 02:11:18.705706 12570 fs_manager.cc:359] Metadata directory not provided
I20260928 02:11:18.705847 12570 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:11:18.706209 12570 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:11:18.709506 12570 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790561432251112-7085-0/minicluster-data/master-0/data/instance:
uuid: "e11399157fa6481585c33e91acb1485f"
format_stamp: "Formatted at 2026-09-28 02:11:18 on dist-test-slave-86gv"
server_key: "bacdc9c0f9b67d2ad89cf962665fe5d4"
server_key_iv: "7c488efb5e218caf90bb0f225de606db"
server_key_version: "encryptionkey@0"
I20260928 02:11:18.710426 12570 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790561432251112-7085-0/minicluster-data/master-0/wal/instance:
uuid: "e11399157fa6481585c33e91acb1485f"
format_stamp: "Formatted at 2026-09-28 02:11:18 on dist-test-slave-86gv"
server_key: "bacdc9c0f9b67d2ad89cf962665fe5d4"
server_key_iv: "7c488efb5e218caf90bb0f225de606db"
server_key_version: "encryptionkey@0"
I20260928 02:11:18.716244 12570 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.004s
I20260928 02:11:18.720078 12584 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:11:18.721634 12570 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 02:11:18.721851 12570 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790561432251112-7085-0/minicluster-data/master-0/data,/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790561432251112-7085-0/minicluster-data/master-0/wal
uuid: "e11399157fa6481585c33e91acb1485f"
format_stamp: "Formatted at 2026-09-28 02:11:18 on dist-test-slave-86gv"
server_key: "bacdc9c0f9b67d2ad89cf962665fe5d4"
server_key_iv: "7c488efb5e218caf90bb0f225de606db"
server_key_version: "encryptionkey@0"
I20260928 02:11:18.722086 12570 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790561432251112-7085-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790561432251112-7085-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790561432251112-7085-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:11:18.745126 12570 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:11:18.746098 12570 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:11:18.746415 12570 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:11:18.765547 12570 rpc_server.cc:307] RPC server started. Bound to: 127.6.235.126:45767
I20260928 02:11:18.765566 12635 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.235.126:45767 every 8 connection(s)
I20260928 02:11:18.767022 12570 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790561432251112-7085-0/minicluster-data/master-0/data/info.pb
I20260928 02:11:18.770905 12636 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:11:18.774259 7085 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTlaUuc/build/asan/bin/kudu as pid 12570
I20260928 02:11:18.774478 7085 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790561432251112-7085-0/minicluster-data/master-0/wal/instance
I20260928 02:11:18.775269 7085 external_mini_cluster.cc:1428] Setting key 90e7e3ead39c5700f2b6d3484c75cffe
I20260928 02:11:18.783561 12636 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e11399157fa6481585c33e91acb1485f: Bootstrap starting.
I20260928 02:11:18.786798 12636 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e11399157fa6481585c33e91acb1485f: Neither blocks nor log segments found. Creating new log.
I20260928 02:11:18.788053 12636 log.cc:824] T 00000000000000000000000000000000 P e11399157fa6481585c33e91acb1485f: Log is configured to *not* fsync() on all Append() calls
I20260928 02:11:18.789077 12639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:11:18.777434 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54372 (local address 127.6.235.126:45767)
0928 02:11:18.778424 (+ 990us) server_negotiation.cc:206] Beginning negotiation
0928 02:11:18.778435 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 02:11:18.778478 (+ 43us) server_negotiation.cc:407] Connection header received
0928 02:11:18.780079 (+ 1601us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:11:18.780103 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:11:18.780616 (+ 513us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:11:18.781054 (+ 438us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:11:18.781678 (+ 624us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:11:18.782941 (+ 1263us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:11:18.784302 (+ 1361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:11:18.785087 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:11:18.785282 (+ 195us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:11:18.786450 (+ 1168us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:11:18.786458 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:11:18.786473 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 02:11:18.787080 (+ 607us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:11:18.787180 (+ 100us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 02:11:18.787204 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:11:18.787363 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0928 02:11:18.787587 (+ 224us) server_negotiation.cc:299] Negotiation successful
0928 02:11:18.787939 (+ 352us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":557,"thread_start_us":329,"threads_started":1}
I20260928 02:11:18.791163 12636 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e11399157fa6481585c33e91acb1485f: No bootstrap required, opened a new log
I20260928 02:11:18.792876 7085 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 02:11:18.799311 12636 raft_consensus.cc:348] T 00000000000000000000000000000000 P e11399157fa6481585c33e91acb1485f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e11399157fa6481585c33e91acb1485f" member_type: VOTER last_known_addr { host: "127.6.235.126" port: 45767 } }
I20260928 02:11:18.799732 12636 raft_consensus.cc:374] T 00000000000000000000000000000000 P e11399157fa6481585c33e91acb1485f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:11:18.799839 12636 raft_consensus.cc:729] T 00000000000000000000000000000000 P e11399157fa6481585c33e91acb1485f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e11399157fa6481585c33e91acb1485f, State: Initialized, Role: FOLLOWER
I20260928 02:11:18.800382 12636 consensus_queue.cc:260] T 00000000000000000000000000000000 P e11399157fa6481585c33e91acb1485f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e11399157fa6481585c33e91acb1485f" member_type: VOTER last_known_addr { host: "127.6.235.126" port: 45767 } }
I20260928 02:11:18.800608 12636 raft_consensus.cc:388] T 00000000000000000000000000000000 P e11399157fa6481585c33e91acb1485f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 02:11:18.800755 12636 raft_consensus.cc:482] T 00000000000000000000000000000000 P e11399157fa6481585c33e91acb1485f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 02:11:18.800972 12636 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e11399157fa6481585c33e91acb1485f [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:11:18.803382 12636 raft_consensus.cc:504] T 00000000000000000000000000000000 P e11399157fa6481585c33e91acb1485f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e11399157fa6481585c33e91acb1485f" member_type: VOTER last_known_addr { host: "127.6.235.126" port: 45767 } }
I20260928 02:11:18.803943 12636 leader_election.cc:302] T 00000000000000000000000000000000 P e11399157fa6481585c33e91acb1485f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e11399157fa6481585c33e91acb1485f; no voters:
I20260928 02:11:18.804587 12636 leader_election.cc:288] T 00000000000000000000000000000000 P e11399157fa6481585c33e91acb1485f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 02:11:18.804759 12641 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e11399157fa6481585c33e91acb1485f [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:11:18.805449 12641 raft_consensus.cc:686] T 00000000000000000000000000000000 P e11399157fa6481585c33e91acb1485f [term 1 LEADER]: Becoming Leader. State: Replica: e11399157fa6481585c33e91acb1485f, State: Running, Role: LEADER
I20260928 02:11:18.806005 12641 consensus_queue.cc:237] T 00000000000000000000000000000000 P e11399157fa6481585c33e91acb1485f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e11399157fa6481585c33e91acb1485f" member_type: VOTER last_known_addr { host: "127.6.235.126" port: 45767 } }
I20260928 02:11:18.806757 12636 sys_catalog.cc:564] T 00000000000000000000000000000000 P e11399157fa6481585c33e91acb1485f [sys.catalog]: configured and running, proceeding with master startup.
I20260928 02:11:18.808934 12642 sys_catalog.cc:455] T 00000000000000000000000000000000 P e11399157fa6481585c33e91acb1485f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e11399157fa6481585c33e91acb1485f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e11399157fa6481585c33e91acb1485f" member_type: VOTER last_known_addr { host: "127.6.235.126" port: 45767 } } }
I20260928 02:11:18.809378 12642 sys_catalog.cc:458] T 00000000000000000000000000000000 P e11399157fa6481585c33e91acb1485f [sys.catalog]: This master's current role is: LEADER
I20260928 02:11:18.809805 12643 sys_catalog.cc:455] T 00000000000000000000000000000000 P e11399157fa6481585c33e91acb1485f [sys.catalog]: SysCatalogTable state changed. Reason: New leader e11399157fa6481585c33e91acb1485f. Latest consensus state: current_term: 1 leader_uuid: "e11399157fa6481585c33e91acb1485f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e11399157fa6481585c33e91acb1485f" member_type: VOTER last_known_addr { host: "127.6.235.126" port: 45767 } } }
I20260928 02:11:18.810261 12643 sys_catalog.cc:458] T 00000000000000000000000000000000 P e11399157fa6481585c33e91acb1485f [sys.catalog]: This master's current role is: LEADER
I20260928 02:11:18.811779 12649 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 02:11:18.817155 12649 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 02:11:18.826478 12649 catalog_manager.cc:1347] Generated new cluster ID: 28bfd455c83d4e42bc5985be12cd4d78
I20260928 02:11:18.826617 12649 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 02:11:18.839787 12649 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 02:11:18.841199 12649 catalog_manager.cc:1504] Loading token signing keys...
I20260928 02:11:18.852456 12649 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e11399157fa6481585c33e91acb1485f: Generated new TSK 0
I20260928 02:11:18.853781 12649 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 02:11:19.339234 12639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:11:18.873010 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54386 (local address 127.6.235.126:45767)
0928 02:11:18.873257 (+ 247us) server_negotiation.cc:206] Beginning negotiation
0928 02:11:18.873265 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 02:11:18.873709 (+ 444us) server_negotiation.cc:407] Connection header received
0928 02:11:18.873939 (+ 230us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:11:18.873943 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:11:18.874020 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:11:18.874148 (+ 128us) server_negotiation.cc:226] Negotiated authn=JWT
0928 02:11:18.874717 (+ 569us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:11:18.875475 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:11:18.877499 (+ 2024us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:11:18.878126 (+ 627us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:11:18.878254 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:11:18.878867 (+ 613us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0928 02:11:19.338669 (+459802us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0928 02:11:19.339012 (+ 343us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:54386: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"server-negotiator.queue_time_us":65}
W20260928 02:11:19.339392 12639 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:54386: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
W20260928 02:11:19.339881 12665 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:11:18.872837 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.235.126:45767 (local address 127.0.0.1:54386)
0928 02:11:18.873573 (+ 736us) negotiation.cc:107] Waiting for socket to connect
0928 02:11:18.873594 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0928 02:11:18.873756 (+ 162us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:11:18.874198 (+ 442us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:11:18.874205 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:11:18.874220 (+ 15us) client_negotiation.cc:189] Negotiated authn=JWT
0928 02:11:18.874567 (+ 347us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:11:18.874580 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:11:18.875617 (+ 1037us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:11:18.875622 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:11:18.877347 (+ 1725us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:11:18.877363 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:11:18.878330 (+ 967us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:11:18.878337 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:11:18.878534 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:11:18.878582 (+ 48us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0928 02:11:19.338956 (+460374us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0928 02:11:19.339185 (+ 229us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.6.235.126:45767: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"client-negotiator.queue_time_us":583,"thread_start_us":377,"threads_started":1}
I20260928 02:11:19.341223 7085 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskTlaUuc/build/asan/bin/kudu pid 12570
I20260928 02:11:19.342514 12631 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 02:11:19.467438 7085 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTlaUuc/build/asan/bin/kudu with pid 12570
2026-09-28T02:11:19Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS (1216 ms)
[ RUN ] SecurityITest.TestWorldReadableKeytab
[ OK ] SecurityITest.TestWorldReadableKeytab (435 ms)
[ RUN ] SecurityITest.TestWorldReadablePrivateKey
[ OK ] SecurityITest.TestWorldReadablePrivateKey (1551 ms)
[ RUN ] SecurityITest.TestCorruptKerberosCC
Loading random data
Initializing database '/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790561432251112-7085-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 02:11:21 dist-test-slave-86gv krb5kdc[12681](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 02:11:21 dist-test-slave-86gv krb5kdc[12681](info): set up 2 sockets
Sep 28 02:11:21 dist-test-slave-86gv krb5kdc[12681](info): commencing operation
krb5kdc: starting...
W20260928 02:11:22.594164 7085 mini_kdc.cc:121] Time spent starting KDC: real 1.099s user 0.002s sys 0.019s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790561432251112-7085-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790561432251112-7085-0/krb5kdc/test-user.keytab.
Sep 28 02:11:22 dist-test-slave-86gv krb5kdc[12681](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561482, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T02:11:22Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T02:11:22Z Disabled control of system clock
WARNING: no policy specified for kudu/127.6.235.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.235.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.235.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790561432251112-7085-0/krb5kdc/kudu_127.6.235.126.keytab.
Entry for principal kudu/127.6.235.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790561432251112-7085-0/krb5kdc/kudu_127.6.235.126.keytab.
I20260928 02:11:22.820952 7085 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTlaUuc/build/asan/bin/kudu
/tmp/dist-test-taskTlaUuc/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790561432251112-7085-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790561432251112-7085-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790561432251112-7085-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790561432251112-7085-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.235.126:37799
--webserver_interface=127.6.235.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790561432251112-7085-0/krb5kdc/kudu_127.6.235.126.keytab
--principal=kudu/127.6.235.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.235.84:35893
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.235.126:37799
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790561432251112-7085-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790561432251112-7085-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790561432251112-7085-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:11:23.150321 12694 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:11:23.150619 12694 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:11:23.150787 12694 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:11:23.158089 12694 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 02:11:23.158211 12694 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:11:23.158329 12694 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:11:23.158430 12694 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 02:11:23.158512 12694 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 02:11:23.168509 12694 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.235.84:35893
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790561432251112-7085-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790561432251112-7085-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.235.126:37799
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790561432251112-7085-0/krb5kdc/kudu_127.6.235.126.keytab
--principal=kudu/127.6.235.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.235.126:37799
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790561432251112-7085-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.235.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790561432251112-7085-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 00:51:59 UTC on ccycloud.quasar-axgkwn.root.comops.site
build id 4676464
ASAN enabled
I20260928 02:11:23.169905 12694 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:11:23.171249 12694 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:11:23.183390 12700 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:11:23.183526 12699 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:11:23.183553 12702 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:11:23.184854 12694 server_base.cc:1034] running on GCE node
I20260928 02:11:23.185784 12694 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:11:23.187362 12694 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:11:23.188594 12694 hybrid_clock.cc:648] HybridClock initialized: now 1790561483188550 us; error 47 us; skew 500 ppm
I20260928 02:11:23.189034 12694 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 02:11:23 dist-test-slave-86gv krb5kdc[12681](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561483, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.235.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:11:23.192183 12694 init.cc:377] Logged in from keytab as kudu/127.6.235.126@KRBTEST.COM (short username kudu)
I20260928 02:11:23.193862 12694 webserver.cc:460] Webserver started at http://127.6.235.126:40445/ using document root <none> and password file <none>
I20260928 02:11:23.194571 12694 fs_manager.cc:359] Metadata directory not provided
I20260928 02:11:23.194737 12694 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:11:23.195084 12694 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:11:23.198194 12694 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790561432251112-7085-0/minicluster-data/master-0/data/instance:
uuid: "9beed66227e54954a71a2ce4ced82923"
format_stamp: "Formatted at 2026-09-28 02:11:23 on dist-test-slave-86gv"
server_key: "871d13f79e4090d396d1daad203fde62"
server_key_iv: "a8e90560fd9457d06ffebc9d7389a0b3"
server_key_version: "encryptionkey@0"
I20260928 02:11:23.198985 12694 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790561432251112-7085-0/minicluster-data/master-0/wal/instance:
uuid: "9beed66227e54954a71a2ce4ced82923"
format_stamp: "Formatted at 2026-09-28 02:11:23 on dist-test-slave-86gv"
server_key: "871d13f79e4090d396d1daad203fde62"
server_key_iv: "a8e90560fd9457d06ffebc9d7389a0b3"
server_key_version: "encryptionkey@0"
I20260928 02:11:23.204456 12694 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20260928 02:11:23.208197 12709 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:11:23.209728 12694 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 02:11:23.209913 12694 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790561432251112-7085-0/minicluster-data/master-0/data,/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790561432251112-7085-0/minicluster-data/master-0/wal
uuid: "9beed66227e54954a71a2ce4ced82923"
format_stamp: "Formatted at 2026-09-28 02:11:23 on dist-test-slave-86gv"
server_key: "871d13f79e4090d396d1daad203fde62"
server_key_iv: "a8e90560fd9457d06ffebc9d7389a0b3"
server_key_version: "encryptionkey@0"
I20260928 02:11:23.210129 12694 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790561432251112-7085-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790561432251112-7085-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790561432251112-7085-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:11:23.227361 12694 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:11:23.231530 12694 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:11:23.231839 12694 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:11:23.251060 12694 rpc_server.cc:307] RPC server started. Bound to: 127.6.235.126:37799
I20260928 02:11:23.251108 12760 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.235.126:37799 every 8 connection(s)
I20260928 02:11:23.252544 12694 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790561432251112-7085-0/minicluster-data/master-0/data/info.pb
I20260928 02:11:23.256608 12761 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:11:23.261706 7085 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTlaUuc/build/asan/bin/kudu as pid 12694
I20260928 02:11:23.261946 7085 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790561432251112-7085-0/minicluster-data/master-0/wal/instance
I20260928 02:11:23.262562 7085 external_mini_cluster.cc:1428] Setting key ad3739ddb46abaf9bcfbf0870a15f448
I20260928 02:11:23.268123 12761 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9beed66227e54954a71a2ce4ced82923: Bootstrap starting.
I20260928 02:11:23.271548 12761 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9beed66227e54954a71a2ce4ced82923: Neither blocks nor log segments found. Creating new log.
I20260928 02:11:23.272650 12761 log.cc:824] T 00000000000000000000000000000000 P 9beed66227e54954a71a2ce4ced82923: Log is configured to *not* fsync() on all Append() calls
I20260928 02:11:23.275556 12761 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9beed66227e54954a71a2ce4ced82923: No bootstrap required, opened a new log
Sep 28 02:11:23 dist-test-slave-86gv krb5kdc[12681](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561482, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.235.126@KRBTEST.COM
I20260928 02:11:23.286170 12761 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9beed66227e54954a71a2ce4ced82923 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9beed66227e54954a71a2ce4ced82923" member_type: VOTER last_known_addr { host: "127.6.235.126" port: 37799 } }
I20260928 02:11:23.286623 12761 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9beed66227e54954a71a2ce4ced82923 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:11:23.286702 12764 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:11:23.264592 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39786 (local address 127.6.235.126:37799)
0928 02:11:23.265451 (+ 859us) server_negotiation.cc:206] Beginning negotiation
0928 02:11:23.265463 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0928 02:11:23.265513 (+ 50us) server_negotiation.cc:407] Connection header received
0928 02:11:23.267266 (+ 1753us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:11:23.267291 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:11:23.267701 (+ 410us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:11:23.268134 (+ 433us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:11:23.269684 (+ 1550us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:11:23.271075 (+ 1391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:11:23.272414 (+ 1339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:11:23.273331 (+ 917us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:11:23.273543 (+ 212us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:11:23.277545 (+ 4002us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:11:23.277555 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:11:23.277575 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:11:23.277621 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:11:23.281880 (+ 4259us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:11:23.282577 (+ 697us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:11:23.282586 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:11:23.282596 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:11:23.282808 (+ 212us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:11:23.283287 (+ 479us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:11:23.283294 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:11:23.283302 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:11:23.284089 (+ 787us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:11:23.284417 (+ 328us) server_negotiation.cc:1035] Waiting for connection context
0928 02:11:23.284793 (+ 376us) server_negotiation.cc:299] Negotiation successful
0928 02:11:23.285304 (+ 511us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":622,"thread_start_us":370,"threads_started":1}
I20260928 02:11:23.286763 12761 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9beed66227e54954a71a2ce4ced82923 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9beed66227e54954a71a2ce4ced82923, State: Initialized, Role: FOLLOWER
I20260928 02:11:23.287395 12761 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9beed66227e54954a71a2ce4ced82923 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9beed66227e54954a71a2ce4ced82923" member_type: VOTER last_known_addr { host: "127.6.235.126" port: 37799 } }
I20260928 02:11:23.287679 12761 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9beed66227e54954a71a2ce4ced82923 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 02:11:23.287840 12761 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9beed66227e54954a71a2ce4ced82923 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 02:11:23.288017 12761 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9beed66227e54954a71a2ce4ced82923 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:11:23.291021 12761 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9beed66227e54954a71a2ce4ced82923 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9beed66227e54954a71a2ce4ced82923" member_type: VOTER last_known_addr { host: "127.6.235.126" port: 37799 } }
I20260928 02:11:23.291561 12761 leader_election.cc:302] T 00000000000000000000000000000000 P 9beed66227e54954a71a2ce4ced82923 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9beed66227e54954a71a2ce4ced82923; no voters:
I20260928 02:11:23.292261 12761 leader_election.cc:288] T 00000000000000000000000000000000 P 9beed66227e54954a71a2ce4ced82923 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 02:11:23.292425 12766 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9beed66227e54954a71a2ce4ced82923 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:11:23.293704 12766 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9beed66227e54954a71a2ce4ced82923 [term 1 LEADER]: Becoming Leader. State: Replica: 9beed66227e54954a71a2ce4ced82923, State: Running, Role: LEADER
I20260928 02:11:23.294294 12766 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9beed66227e54954a71a2ce4ced82923 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9beed66227e54954a71a2ce4ced82923" member_type: VOTER last_known_addr { host: "127.6.235.126" port: 37799 } }
I20260928 02:11:23.294555 12761 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9beed66227e54954a71a2ce4ced82923 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 02:11:23.297659 12767 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9beed66227e54954a71a2ce4ced82923 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9beed66227e54954a71a2ce4ced82923" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9beed66227e54954a71a2ce4ced82923" member_type: VOTER last_known_addr { host: "127.6.235.126" port: 37799 } } }
I20260928 02:11:23.297631 12768 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9beed66227e54954a71a2ce4ced82923 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9beed66227e54954a71a2ce4ced82923. Latest consensus state: current_term: 1 leader_uuid: "9beed66227e54954a71a2ce4ced82923" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9beed66227e54954a71a2ce4ced82923" member_type: VOTER last_known_addr { host: "127.6.235.126" port: 37799 } } }
I20260928 02:11:23.298137 12768 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9beed66227e54954a71a2ce4ced82923 [sys.catalog]: This master's current role is: LEADER
I20260928 02:11:23.298137 12767 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9beed66227e54954a71a2ce4ced82923 [sys.catalog]: This master's current role is: LEADER
I20260928 02:11:23.299816 12775 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 02:11:23.306500 12775 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 02:11:23.317466 12775 catalog_manager.cc:1347] Generated new cluster ID: 24755bf0ab784cf39b4f23e359b11c27
I20260928 02:11:23.317603 12775 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 02:11:23.337102 12775 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 02:11:23.338707 12775 catalog_manager.cc:1504] Loading token signing keys...
I20260928 02:11:23.349891 12775 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9beed66227e54954a71a2ce4ced82923: Generated new TSK 0
I20260928 02:11:23.351104 12775 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.6.235.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.235.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.235.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790561432251112-7085-0/krb5kdc/kudu_127.6.235.65.keytab.
Entry for principal kudu/127.6.235.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790561432251112-7085-0/krb5kdc/kudu_127.6.235.65.keytab.
I20260928 02:11:23.417165 7085 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTlaUuc/build/asan/bin/kudu
/tmp/dist-test-taskTlaUuc/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790561432251112-7085-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790561432251112-7085-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790561432251112-7085-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790561432251112-7085-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.235.65:0
--local_ip_for_outbound_sockets=127.6.235.65
--webserver_interface=127.6.235.65
--webserver_port=0
--tserver_master_addrs=127.6.235.126:37799
--keytab_file=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790561432251112-7085-0/krb5kdc/kudu_127.6.235.65.keytab
--principal=kudu/127.6.235.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.235.84:35893
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790561432251112-7085-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790561432251112-7085-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790561432251112-7085-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:11:23.754757 12787 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:11:23.755066 12787 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:11:23.755163 12787 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:11:23.762603 12787 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:11:23.762723 12787 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:11:23.762835 12787 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.235.65
I20260928 02:11:23.772634 12787 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.235.84:35893
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790561432251112-7085-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790561432251112-7085-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790561432251112-7085-0/krb5kdc/kudu_127.6.235.65.keytab
--principal=kudu/127.6.235.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.235.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790561432251112-7085-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.6.235.65
--webserver_port=0
--tserver_master_addrs=127.6.235.126:37799
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.235.65
--log_dir=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790561432251112-7085-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 00:51:59 UTC on ccycloud.quasar-axgkwn.root.comops.site
build id 4676464
ASAN enabled
I20260928 02:11:23.774000 12787 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:11:23.775410 12787 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:11:23.788792 12792 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:11:23.788926 12795 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:11:23.788862 12793 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:11:23.789863 12787 server_base.cc:1034] running on GCE node
I20260928 02:11:23.790510 12787 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:11:23.791512 12787 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:11:23.792711 12787 hybrid_clock.cc:648] HybridClock initialized: now 1790561483792637 us; error 77 us; skew 500 ppm
I20260928 02:11:23.793080 12787 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 02:11:23 dist-test-slave-86gv krb5kdc[12681](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561483, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.235.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:11:23.796563 12787 init.cc:377] Logged in from keytab as kudu/127.6.235.65@KRBTEST.COM (short username kudu)
I20260928 02:11:23.797986 12787 webserver.cc:460] Webserver started at http://127.6.235.65:37861/ using document root <none> and password file <none>
I20260928 02:11:23.798640 12787 fs_manager.cc:359] Metadata directory not provided
I20260928 02:11:23.798735 12787 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:11:23.799021 12787 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:11:23.802107 12787 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790561432251112-7085-0/minicluster-data/ts-0/data/instance:
uuid: "94254866d06947fe88949124a76ada9f"
format_stamp: "Formatted at 2026-09-28 02:11:23 on dist-test-slave-86gv"
server_key: "eddce7e1c615ba8f63c4e4d84f7b7091"
server_key_iv: "454ebde5460e8147a527d2ff45947b4d"
server_key_version: "encryptionkey@0"
I20260928 02:11:23.802929 12787 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790561432251112-7085-0/minicluster-data/ts-0/wal/instance:
uuid: "94254866d06947fe88949124a76ada9f"
format_stamp: "Formatted at 2026-09-28 02:11:23 on dist-test-slave-86gv"
server_key: "eddce7e1c615ba8f63c4e4d84f7b7091"
server_key_iv: "454ebde5460e8147a527d2ff45947b4d"
server_key_version: "encryptionkey@0"
I20260928 02:11:23.808166 12787 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260928 02:11:23.811863 12802 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:11:23.813432 12787 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 02:11:23.813647 12787 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790561432251112-7085-0/minicluster-data/ts-0/data,/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790561432251112-7085-0/minicluster-data/ts-0/wal
uuid: "94254866d06947fe88949124a76ada9f"
format_stamp: "Formatted at 2026-09-28 02:11:23 on dist-test-slave-86gv"
server_key: "eddce7e1c615ba8f63c4e4d84f7b7091"
server_key_iv: "454ebde5460e8147a527d2ff45947b4d"
server_key_version: "encryptionkey@0"
I20260928 02:11:23.813961 12787 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790561432251112-7085-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790561432251112-7085-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790561432251112-7085-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:11:23.832288 12787 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:11:23.837010 12787 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:11:23.837324 12787 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:11:23.838395 12787 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:11:23.839892 12787 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:11:23.840018 12787 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:11:23.840179 12787 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:11:23.840253 12787 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:11:23.874948 12787 rpc_server.cc:307] RPC server started. Bound to: 127.6.235.65:43179
I20260928 02:11:23.874989 12914 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.235.65:43179 every 8 connection(s)
I20260928 02:11:23.876304 12787 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790561432251112-7085-0/minicluster-data/ts-0/data/info.pb
I20260928 02:11:23.879004 7085 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTlaUuc/build/asan/bin/kudu as pid 12787
I20260928 02:11:23.879251 7085 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790561432251112-7085-0/minicluster-data/ts-0/wal/instance
I20260928 02:11:23.880016 7085 external_mini_cluster.cc:1428] Setting key c7f6cdcbec3f90a549eecef265515abb
Sep 28 02:11:23 dist-test-slave-86gv krb5kdc[12681](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561483, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.235.65@KRBTEST.COM for kudu/127.6.235.126@KRBTEST.COM
I20260928 02:11:23.900686 12917 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:11:23.879135 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.235.65:58413 (local address 127.6.235.126:37799)
0928 02:11:23.879694 (+ 559us) server_negotiation.cc:206] Beginning negotiation
0928 02:11:23.879704 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 02:11:23.880731 (+ 1027us) server_negotiation.cc:407] Connection header received
0928 02:11:23.882898 (+ 2167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:11:23.882902 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:11:23.882963 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:11:23.883055 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:11:23.885303 (+ 2248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:11:23.886036 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:11:23.888542 (+ 2506us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:11:23.889021 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:11:23.889190 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:11:23.893277 (+ 4087us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:11:23.893286 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:11:23.893293 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:11:23.893339 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:11:23.896999 (+ 3660us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:11:23.897802 (+ 803us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:11:23.897810 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:11:23.897817 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:11:23.897969 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:11:23.898408 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:11:23.898417 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:11:23.898423 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:11:23.898827 (+ 404us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:11:23.898981 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0928 02:11:23.899910 (+ 929us) server_negotiation.cc:299] Negotiation successful
0928 02:11:23.900168 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":416,"thread_start_us":292,"threads_started":1}
I20260928 02:11:23.901521 12918 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:11:23.879704 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.235.126:37799 (local address 127.6.235.65:58413)
0928 02:11:23.880538 (+ 834us) negotiation.cc:107] Waiting for socket to connect
0928 02:11:23.880589 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0928 02:11:23.882583 (+ 1994us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:11:23.883356 (+ 773us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:11:23.883371 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:11:23.884197 (+ 826us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:11:23.885075 (+ 878us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:11:23.885106 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:11:23.886207 (+ 1101us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:11:23.886216 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:11:23.888386 (+ 2170us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:11:23.888404 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:11:23.889277 (+ 873us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:11:23.889283 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:11:23.889488 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:11:23.890348 (+ 860us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:11:23.890376 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:11:23.892973 (+ 2597us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:11:23.897148 (+ 4175us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:11:23.897156 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:11:23.897171 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:11:23.897645 (+ 474us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:11:23.898075 (+ 430us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:11:23.898078 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:11:23.898080 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:11:23.898280 (+ 200us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:11:23.898945 (+ 665us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:11:23.898975 (+ 30us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:11:23.899432 (+ 457us) client_negotiation.cc:769] Sending connection context
0928 02:11:23.899741 (+ 309us) client_negotiation.cc:240] Negotiation successful
0928 02:11:23.900246 (+ 505us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":562,"thread_start_us":345,"threads_started":1}
I20260928 02:11:23.903179 12915 heartbeater.cc:347] Connected to a master server at 127.6.235.126:37799
I20260928 02:11:23.903512 12915 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.6.235.66@KRBTEST.COM; defaulting to no policy
I20260928 02:11:23.904332 12915 heartbeater.cc:511] Master 127.6.235.126:37799 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.235.66@KRBTEST.COM" created.
I20260928 02:11:23.906538 12726 ts_manager.cc:194] Registered new tserver with Master: 94254866d06947fe88949124a76ada9f (127.6.235.65:43179)
I20260928 02:11:23.908020 12726 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.235.65@KRBTEST.COM'} at 127.6.235.65:58413
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.235.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790561432251112-7085-0/krb5kdc/kudu_127.6.235.66.keytab.
Entry for principal kudu/127.6.235.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790561432251112-7085-0/krb5kdc/kudu_127.6.235.66.keytab.
I20260928 02:11:23.932015 7085 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTlaUuc/build/asan/bin/kudu
/tmp/dist-test-taskTlaUuc/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790561432251112-7085-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790561432251112-7085-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790561432251112-7085-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790561432251112-7085-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.235.66:0
--local_ip_for_outbound_sockets=127.6.235.66
--webserver_interface=127.6.235.66
--webserver_port=0
--tserver_master_addrs=127.6.235.126:37799
--keytab_file=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790561432251112-7085-0/krb5kdc/kudu_127.6.235.66.keytab
--principal=kudu/127.6.235.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.235.84:35893
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790561432251112-7085-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790561432251112-7085-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790561432251112-7085-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:11:24.267863 12921 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:11:24.268164 12921 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:11:24.268258 12921 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:11:24.275763 12921 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:11:24.275882 12921 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:11:24.275996 12921 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.235.66
I20260928 02:11:24.286044 12921 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.235.84:35893
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790561432251112-7085-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790561432251112-7085-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790561432251112-7085-0/krb5kdc/kudu_127.6.235.66.keytab
--principal=kudu/127.6.235.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.235.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790561432251112-7085-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.6.235.66
--webserver_port=0
--tserver_master_addrs=127.6.235.126:37799
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.235.66
--log_dir=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790561432251112-7085-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 00:51:59 UTC on ccycloud.quasar-axgkwn.root.comops.site
build id 4676464
ASAN enabled
I20260928 02:11:24.287317 12921 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:11:24.288594 12921 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:11:24.302587 12926 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:11:24.302650 12929 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:11:24.302583 12927 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:11:24.303349 12921 server_base.cc:1034] running on GCE node
I20260928 02:11:24.304000 12921 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:11:24.305056 12921 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:11:24.306301 12921 hybrid_clock.cc:648] HybridClock initialized: now 1790561484306243 us; error 65 us; skew 500 ppm
I20260928 02:11:24.306638 12921 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 02:11:24 dist-test-slave-86gv krb5kdc[12681](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561484, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.235.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:11:24.309875 12921 init.cc:377] Logged in from keytab as kudu/127.6.235.66@KRBTEST.COM (short username kudu)
I20260928 02:11:24.311538 12921 webserver.cc:460] Webserver started at http://127.6.235.66:34637/ using document root <none> and password file <none>
I20260928 02:11:24.312273 12921 fs_manager.cc:359] Metadata directory not provided
I20260928 02:11:24.312422 12921 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:11:24.312734 12921 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:11:24.315948 12921 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790561432251112-7085-0/minicluster-data/ts-1/data/instance:
uuid: "d1561eab569546a9908c781c70bf7c67"
format_stamp: "Formatted at 2026-09-28 02:11:24 on dist-test-slave-86gv"
server_key: "43e188614a3a0b61316f926b8ef7142c"
server_key_iv: "fb5caaf8590c0205808443af35f2bc6a"
server_key_version: "encryptionkey@0"
I20260928 02:11:24.316776 12921 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790561432251112-7085-0/minicluster-data/ts-1/wal/instance:
uuid: "d1561eab569546a9908c781c70bf7c67"
format_stamp: "Formatted at 2026-09-28 02:11:24 on dist-test-slave-86gv"
server_key: "43e188614a3a0b61316f926b8ef7142c"
server_key_iv: "fb5caaf8590c0205808443af35f2bc6a"
server_key_version: "encryptionkey@0"
I20260928 02:11:24.322090 12921 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260928 02:11:24.325680 12936 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:11:24.327050 12921 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260928 02:11:24.327235 12921 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790561432251112-7085-0/minicluster-data/ts-1/data,/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790561432251112-7085-0/minicluster-data/ts-1/wal
uuid: "d1561eab569546a9908c781c70bf7c67"
format_stamp: "Formatted at 2026-09-28 02:11:24 on dist-test-slave-86gv"
server_key: "43e188614a3a0b61316f926b8ef7142c"
server_key_iv: "fb5caaf8590c0205808443af35f2bc6a"
server_key_version: "encryptionkey@0"
I20260928 02:11:24.327459 12921 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790561432251112-7085-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790561432251112-7085-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790561432251112-7085-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:11:24.363427 12921 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:11:24.368311 12921 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:11:24.368630 12921 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:11:24.369793 12921 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:11:24.371220 12921 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:11:24.371328 12921 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:11:24.371487 12921 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:11:24.371562 12921 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:11:24.405735 12921 rpc_server.cc:307] RPC server started. Bound to: 127.6.235.66:41785
I20260928 02:11:24.405786 13048 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.235.66:41785 every 8 connection(s)
I20260928 02:11:24.407128 12921 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790561432251112-7085-0/minicluster-data/ts-1/data/info.pb
I20260928 02:11:24.414579 7085 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTlaUuc/build/asan/bin/kudu as pid 12921
I20260928 02:11:24.414793 7085 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790561432251112-7085-0/minicluster-data/ts-1/wal/instance
I20260928 02:11:24.415222 7085 external_mini_cluster.cc:1428] Setting key 69cba24b6010214b1b45b841a4dd3e06
Sep 28 02:11:24 dist-test-slave-86gv krb5kdc[12681](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561484, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.235.66@KRBTEST.COM for kudu/127.6.235.126@KRBTEST.COM
I20260928 02:11:24.428056 13051 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:11:24.409864 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.235.66:39155 (local address 127.6.235.126:37799)
0928 02:11:24.410256 (+ 392us) server_negotiation.cc:206] Beginning negotiation
0928 02:11:24.410263 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 02:11:24.411295 (+ 1032us) server_negotiation.cc:407] Connection header received
0928 02:11:24.413101 (+ 1806us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:11:24.413107 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:11:24.413198 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:11:24.413319 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:11:24.415047 (+ 1728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:11:24.415798 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:11:24.417765 (+ 1967us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:11:24.418177 (+ 412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:11:24.418314 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:11:24.421856 (+ 3542us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:11:24.421865 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:11:24.421872 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:11:24.421919 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:11:24.424789 (+ 2870us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:11:24.425782 (+ 993us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:11:24.425785 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:11:24.425788 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:11:24.425906 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:11:24.426386 (+ 480us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:11:24.426389 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:11:24.426391 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:11:24.426628 (+ 237us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:11:24.426748 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0928 02:11:24.427619 (+ 871us) server_negotiation.cc:299] Negotiation successful
0928 02:11:24.427799 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":288,"thread_start_us":195,"threads_started":1}
I20260928 02:11:24.429656 13052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:11:24.410443 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.235.126:37799 (local address 127.6.235.66:39155)
0928 02:11:24.411128 (+ 685us) negotiation.cc:107] Waiting for socket to connect
0928 02:11:24.411167 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0928 02:11:24.412783 (+ 1616us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:11:24.413569 (+ 786us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:11:24.413578 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:11:24.414121 (+ 543us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:11:24.414864 (+ 743us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:11:24.414892 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:11:24.415950 (+ 1058us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:11:24.415955 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:11:24.417616 (+ 1661us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:11:24.417628 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:11:24.418410 (+ 782us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:11:24.418417 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:11:24.418699 (+ 282us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:11:24.419417 (+ 718us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:11:24.419444 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:11:24.421618 (+ 2174us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:11:24.424969 (+ 3351us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:11:24.424981 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:11:24.425001 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:11:24.425640 (+ 639us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:11:24.426012 (+ 372us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:11:24.426019 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:11:24.426026 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:11:24.426274 (+ 248us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:11:24.426763 (+ 489us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:11:24.426774 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:11:24.427242 (+ 468us) client_negotiation.cc:769] Sending connection context
0928 02:11:24.427537 (+ 295us) client_negotiation.cc:240] Negotiation successful
0928 02:11:24.428123 (+ 586us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":458,"thread_start_us":280,"threads_started":1}
I20260928 02:11:24.431245 13049 heartbeater.cc:347] Connected to a master server at 127.6.235.126:37799
I20260928 02:11:24.431584 13049 heartbeater.cc:464] Registering TS with master...
I20260928 02:11:24.432519 13049 heartbeater.cc:511] Master 127.6.235.126:37799 requested a full tablet report, sending...
I20260928 02:11:24.434185 12726 ts_manager.cc:194] Registered new tserver with Master: d1561eab569546a9908c781c70bf7c67 (127.6.235.66:41785)
I20260928 02:11:24.435065 12726 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.235.66@KRBTEST.COM'} at 127.6.235.66:39155
WARNING: no policy specified for kudu/127.6.235.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.235.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.235.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790561432251112-7085-0/krb5kdc/kudu_127.6.235.67.keytab.
Entry for principal kudu/127.6.235.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790561432251112-7085-0/krb5kdc/kudu_127.6.235.67.keytab.
I20260928 02:11:24.468575 7085 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTlaUuc/build/asan/bin/kudu
/tmp/dist-test-taskTlaUuc/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790561432251112-7085-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790561432251112-7085-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790561432251112-7085-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790561432251112-7085-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.235.67:0
--local_ip_for_outbound_sockets=127.6.235.67
--webserver_interface=127.6.235.67
--webserver_port=0
--tserver_master_addrs=127.6.235.126:37799
--keytab_file=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790561432251112-7085-0/krb5kdc/kudu_127.6.235.67.keytab
--principal=kudu/127.6.235.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.235.84:35893
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790561432251112-7085-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790561432251112-7085-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790561432251112-7085-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:11:24.809479 13055 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:11:24.809762 13055 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:11:24.809854 13055 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:11:24.817615 13055 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:11:24.817731 13055 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:11:24.817909 13055 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.235.67
I20260928 02:11:24.828326 13055 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.235.84:35893
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790561432251112-7085-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790561432251112-7085-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790561432251112-7085-0/krb5kdc/kudu_127.6.235.67.keytab
--principal=kudu/127.6.235.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.235.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790561432251112-7085-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.6.235.67
--webserver_port=0
--tserver_master_addrs=127.6.235.126:37799
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.235.67
--log_dir=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790561432251112-7085-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 00:51:59 UTC on ccycloud.quasar-axgkwn.root.comops.site
build id 4676464
ASAN enabled
I20260928 02:11:24.829717 13055 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:11:24.831113 13055 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:11:24.845290 13060 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:11:24.845180 13061 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:11:24.845278 13063 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:11:24.846190 13055 server_base.cc:1034] running on GCE node
I20260928 02:11:24.846889 13055 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:11:24.847950 13055 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:11:24.849192 13055 hybrid_clock.cc:648] HybridClock initialized: now 1790561484849136 us; error 53 us; skew 500 ppm
I20260928 02:11:24.849566 13055 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 02:11:24 dist-test-slave-86gv krb5kdc[12681](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561484, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.235.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:11:24.853065 13055 init.cc:377] Logged in from keytab as kudu/127.6.235.67@KRBTEST.COM (short username kudu)
I20260928 02:11:24.854696 13055 webserver.cc:460] Webserver started at http://127.6.235.67:41669/ using document root <none> and password file <none>
I20260928 02:11:24.855439 13055 fs_manager.cc:359] Metadata directory not provided
I20260928 02:11:24.855548 13055 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:11:24.855825 13055 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:11:24.859073 13055 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790561432251112-7085-0/minicluster-data/ts-2/data/instance:
uuid: "dc9a92824769487f919a215b6d4faae6"
format_stamp: "Formatted at 2026-09-28 02:11:24 on dist-test-slave-86gv"
server_key: "ffd6e49567d042c00a7da55fd8e2c297"
server_key_iv: "88690eb4c94b42f8e9a5db9c57b656ca"
server_key_version: "encryptionkey@0"
I20260928 02:11:24.859918 13055 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790561432251112-7085-0/minicluster-data/ts-2/wal/instance:
uuid: "dc9a92824769487f919a215b6d4faae6"
format_stamp: "Formatted at 2026-09-28 02:11:24 on dist-test-slave-86gv"
server_key: "ffd6e49567d042c00a7da55fd8e2c297"
server_key_iv: "88690eb4c94b42f8e9a5db9c57b656ca"
server_key_version: "encryptionkey@0"
I20260928 02:11:24.865892 13055 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.000s
I20260928 02:11:24.869678 13070 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:11:24.871095 13055 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260928 02:11:24.871245 13055 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790561432251112-7085-0/minicluster-data/ts-2/data,/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790561432251112-7085-0/minicluster-data/ts-2/wal
uuid: "dc9a92824769487f919a215b6d4faae6"
format_stamp: "Formatted at 2026-09-28 02:11:24 on dist-test-slave-86gv"
server_key: "ffd6e49567d042c00a7da55fd8e2c297"
server_key_iv: "88690eb4c94b42f8e9a5db9c57b656ca"
server_key_version: "encryptionkey@0"
I20260928 02:11:24.871470 13055 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790561432251112-7085-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790561432251112-7085-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790561432251112-7085-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:11:24.887506 13055 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:11:24.891963 13055 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:11:24.892299 13055 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:11:24.893349 13055 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:11:24.894618 13055 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:11:24.894688 13055 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:11:24.894776 13055 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:11:24.894826 13055 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:11:24.911154 12915 heartbeater.cc:503] Master 127.6.235.126:37799 was elected leader, sending a full tablet report...
I20260928 02:11:24.929838 13055 rpc_server.cc:307] RPC server started. Bound to: 127.6.235.67:46789
I20260928 02:11:24.929874 13183 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.235.67:46789 every 8 connection(s)
I20260928 02:11:24.931489 13055 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790561432251112-7085-0/minicluster-data/ts-2/data/info.pb
I20260928 02:11:24.941102 7085 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTlaUuc/build/asan/bin/kudu as pid 13055
I20260928 02:11:24.941308 7085 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790561432251112-7085-0/minicluster-data/ts-2/wal/instance
I20260928 02:11:24.941862 7085 external_mini_cluster.cc:1428] Setting key d5fccebf4dfa68ea20578f75f2c8e8bd
Sep 28 02:11:24 dist-test-slave-86gv krb5kdc[12681](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561484, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.235.67@KRBTEST.COM for kudu/127.6.235.126@KRBTEST.COM
I20260928 02:11:24.952081 13186 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:11:24.934323 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.235.67:46711 (local address 127.6.235.126:37799)
0928 02:11:24.934806 (+ 483us) server_negotiation.cc:206] Beginning negotiation
0928 02:11:24.934813 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 02:11:24.935828 (+ 1015us) server_negotiation.cc:407] Connection header received
0928 02:11:24.937622 (+ 1794us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:11:24.937626 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:11:24.937702 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:11:24.937801 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:11:24.939299 (+ 1498us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:11:24.940047 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:11:24.942176 (+ 2129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:11:24.942611 (+ 435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:11:24.942743 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:11:24.946720 (+ 3977us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:11:24.946724 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:11:24.946727 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:11:24.946764 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:11:24.949465 (+ 2701us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:11:24.950137 (+ 672us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:11:24.950140 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:11:24.950143 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:11:24.950276 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:11:24.950638 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:11:24.950641 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:11:24.950643 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:11:24.950891 (+ 248us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:11:24.951006 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0928 02:11:24.951659 (+ 653us) server_negotiation.cc:299] Negotiation successful
0928 02:11:24.951820 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":355,"thread_start_us":220,"threads_started":1}
I20260928 02:11:24.953172 13187 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:11:24.934920 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.235.126:37799 (local address 127.6.235.67:46711)
0928 02:11:24.935655 (+ 735us) negotiation.cc:107] Waiting for socket to connect
0928 02:11:24.935698 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0928 02:11:24.937363 (+ 1665us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:11:24.938008 (+ 645us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:11:24.938016 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:11:24.938555 (+ 539us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:11:24.939128 (+ 573us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:11:24.939145 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:11:24.940173 (+ 1028us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:11:24.940177 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:11:24.942029 (+ 1852us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:11:24.942043 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:11:24.942821 (+ 778us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:11:24.942826 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:11:24.943181 (+ 355us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:11:24.943989 (+ 808us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:11:24.944011 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:11:24.946539 (+ 2528us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:11:24.949587 (+ 3048us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:11:24.949594 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:11:24.949608 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:11:24.950026 (+ 418us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:11:24.950366 (+ 340us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:11:24.950370 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:11:24.950372 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:11:24.950543 (+ 171us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:11:24.950996 (+ 453us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:11:24.951001 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:11:24.951351 (+ 350us) client_negotiation.cc:769] Sending connection context
0928 02:11:24.951563 (+ 212us) client_negotiation.cc:240] Negotiation successful
0928 02:11:24.951937 (+ 374us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":503,"thread_start_us":311,"threads_started":1}
I20260928 02:11:24.954576 13184 heartbeater.cc:347] Connected to a master server at 127.6.235.126:37799
I20260928 02:11:24.954893 13184 heartbeater.cc:464] Registering TS with master...
I20260928 02:11:24.955690 13184 heartbeater.cc:511] Master 127.6.235.126:37799 requested a full tablet report, sending...
I20260928 02:11:24.957396 12726 ts_manager.cc:194] Registered new tserver with Master: dc9a92824769487f919a215b6d4faae6 (127.6.235.67:46789)
I20260928 02:11:24.958297 12726 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.235.67@KRBTEST.COM'} at 127.6.235.67:46711
I20260928 02:11:24.970062 7085 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal test-admin with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790561432251112-7085-0/krb5kdc/test-admin.keytab.
Entry for principal test-admin with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790561432251112-7085-0/krb5kdc/test-admin.keytab.
Sep 28 02:11:24 dist-test-slave-86gv krb5kdc[12681](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561484, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:11:25.000383 7085 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
W20260928 02:11:25.005672 13192 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:11:25.003148 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59542 (local address 127.6.235.65:43179)
0928 02:11:25.003849 (+ 701us) server_negotiation.cc:206] Beginning negotiation
0928 02:11:25.003857 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 02:11:25.003885 (+ 28us) server_negotiation.cc:407] Connection header received
0928 02:11:25.003946 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:11:25.003952 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:11:25.004167 (+ 215us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:11:25.004304 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:11:25.005251 (+ 947us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59542: BlockingRecv error: recv got EOF from 127.0.0.1:59542 (error 108)
Metrics: {"server-negotiator.queue_time_us":534,"thread_start_us":377,"threads_started":1}
W20260928 02:11:25.006122 7085 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790561432251112-7085-0/krb5kdc/krb5cc' of type FILE
Sep 28 02:11:25 dist-test-slave-86gv krb5kdc[12681](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561485, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:11:25.007673 7085 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 28 02:11:25 dist-test-slave-86gv krb5kdc[12681](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561485, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:11:25.009070 7085 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 28 02:11:25 dist-test-slave-86gv krb5kdc[12681](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561485, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.235.65@KRBTEST.COM
I20260928 02:11:25.024261 13192 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:11:25.010686 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59554 (local address 127.6.235.65:43179)
0928 02:11:25.010895 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0928 02:11:25.010901 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 02:11:25.011111 (+ 210us) server_negotiation.cc:407] Connection header received
0928 02:11:25.011235 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:11:25.011238 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:11:25.011310 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:11:25.011392 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:11:25.012350 (+ 958us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:11:25.013421 (+ 1071us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:11:25.014663 (+ 1242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:11:25.015325 (+ 662us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:11:25.015423 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:11:25.018973 (+ 3550us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:11:25.018984 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:11:25.019000 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:11:25.019034 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:11:25.021825 (+ 2791us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:11:25.022458 (+ 633us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:11:25.022465 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:11:25.022468 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:11:25.022578 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:11:25.022882 (+ 304us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:11:25.022885 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:11:25.022887 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:11:25.023228 (+ 341us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:11:25.023435 (+ 207us) server_negotiation.cc:1035] Waiting for connection context
0928 02:11:25.023710 (+ 275us) server_negotiation.cc:299] Negotiation successful
0928 02:11:25.023973 (+ 263us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
W20260928 02:11:25.029897 13192 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:11:25.028357 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59560 (local address 127.6.235.65:43179)
0928 02:11:25.028555 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0928 02:11:25.028559 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:11:25.028652 (+ 93us) server_negotiation.cc:407] Connection header received
0928 02:11:25.028795 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:11:25.028798 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:11:25.028862 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:11:25.028999 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:11:25.029650 (+ 651us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59560: BlockingRecv error: recv got EOF from 127.0.0.1:59560 (error 108)
Metrics: {"server-negotiator.queue_time_us":68}
W20260928 02:11:25.030006 7085 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790561432251112-7085-0/krb5kdc/krb5cc' of type FILE
Sep 28 02:11:25 dist-test-slave-86gv krb5kdc[12681](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561485, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:11:25.031622 7085 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 28 02:11:25 dist-test-slave-86gv krb5kdc[12681](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561485, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:11:25.033039 7085 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 28 02:11:25 dist-test-slave-86gv krb5kdc[12681](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561485, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.235.65@KRBTEST.COM
I20260928 02:11:25.047605 13192 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:11:25.034297 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59570 (local address 127.6.235.65:43179)
0928 02:11:25.034560 (+ 263us) server_negotiation.cc:206] Beginning negotiation
0928 02:11:25.034567 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 02:11:25.034828 (+ 261us) server_negotiation.cc:407] Connection header received
0928 02:11:25.035104 (+ 276us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:11:25.035110 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:11:25.035173 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:11:25.035299 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:11:25.036276 (+ 977us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:11:25.037042 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:11:25.038287 (+ 1245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:11:25.038735 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:11:25.038874 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:11:25.042564 (+ 3690us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:11:25.042577 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:11:25.042581 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:11:25.042625 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:11:25.045030 (+ 2405us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:11:25.045691 (+ 661us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:11:25.045698 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:11:25.045702 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:11:25.045835 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:11:25.046204 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:11:25.046209 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:11:25.046221 (+ 12us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:11:25.046491 (+ 270us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:11:25.046751 (+ 260us) server_negotiation.cc:1035] Waiting for connection context
0928 02:11:25.047081 (+ 330us) server_negotiation.cc:299] Negotiation successful
0928 02:11:25.047290 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":97}
W20260928 02:11:25.052650 13192 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:11:25.050730 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59576 (local address 127.6.235.65:43179)
0928 02:11:25.050929 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0928 02:11:25.050937 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 02:11:25.051316 (+ 379us) server_negotiation.cc:407] Connection header received
0928 02:11:25.051529 (+ 213us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:11:25.051534 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:11:25.051606 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:11:25.051706 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:11:25.052415 (+ 709us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59576: BlockingRecv error: recv got EOF from 127.0.0.1:59576 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
W20260928 02:11:25.052982 7085 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790561432251112-7085-0/krb5kdc/krb5cc' of type FILE
Sep 28 02:11:25 dist-test-slave-86gv krb5kdc[12681](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561485, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:11:25.054544 7085 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 28 02:11:25 dist-test-slave-86gv krb5kdc[12681](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561485, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:11:25.055958 7085 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 28 02:11:25 dist-test-slave-86gv krb5kdc[12681](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561485, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.235.65@KRBTEST.COM
I20260928 02:11:25.069697 13192 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:11:25.057309 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59578 (local address 127.6.235.65:43179)
0928 02:11:25.057534 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0928 02:11:25.057540 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 02:11:25.057721 (+ 181us) server_negotiation.cc:407] Connection header received
0928 02:11:25.057874 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:11:25.057877 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:11:25.057934 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:11:25.058014 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:11:25.058901 (+ 887us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:11:25.059582 (+ 681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:11:25.060776 (+ 1194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:11:25.061357 (+ 581us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:11:25.061495 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:11:25.065113 (+ 3618us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:11:25.065118 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:11:25.065122 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:11:25.065154 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:11:25.067537 (+ 2383us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:11:25.068146 (+ 609us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:11:25.068150 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:11:25.068152 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:11:25.068270 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:11:25.068554 (+ 284us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:11:25.068557 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:11:25.068559 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:11:25.068833 (+ 274us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:11:25.069128 (+ 295us) server_negotiation.cc:1035] Waiting for connection context
0928 02:11:25.069280 (+ 152us) server_negotiation.cc:299] Negotiation successful
0928 02:11:25.069436 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260928 02:11:25.071362 7085 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskTlaUuc/build/asan/bin/kudu pid 12787
I20260928 02:11:25.082621 13192 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:11:25.072050 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59592 (local address 127.6.235.65:43179)
0928 02:11:25.072234 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0928 02:11:25.072242 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 02:11:25.072557 (+ 315us) server_negotiation.cc:407] Connection header received
0928 02:11:25.072658 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:11:25.072661 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:11:25.072718 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:11:25.072803 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:11:25.073757 (+ 954us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:11:25.074431 (+ 674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:11:25.075648 (+ 1217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:11:25.076054 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:11:25.076150 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:11:25.078159 (+ 2009us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:11:25.078163 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:11:25.078166 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:11:25.078194 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:11:25.080377 (+ 2183us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:11:25.080988 (+ 611us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:11:25.080992 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:11:25.080994 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:11:25.081111 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:11:25.081418 (+ 307us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:11:25.081421 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:11:25.081423 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:11:25.081654 (+ 231us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:11:25.081837 (+ 183us) server_negotiation.cc:1035] Waiting for connection context
0928 02:11:25.082104 (+ 267us) server_negotiation.cc:299] Negotiation successful
0928 02:11:25.082255 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260928 02:11:25.083243 12910 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 02:11:25.167883 7085 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTlaUuc/build/asan/bin/kudu with pid 12787
I20260928 02:11:25.185240 7085 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskTlaUuc/build/asan/bin/kudu pid 12921
Sep 28 02:11:25 dist-test-slave-86gv krb5kdc[12681](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561485, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.235.66@KRBTEST.COM
I20260928 02:11:25.201139 13205 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:11:25.186099 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47808 (local address 127.6.235.66:41785)
0928 02:11:25.186698 (+ 599us) server_negotiation.cc:206] Beginning negotiation
0928 02:11:25.186705 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 02:11:25.186738 (+ 33us) server_negotiation.cc:407] Connection header received
0928 02:11:25.186800 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:11:25.186806 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:11:25.187020 (+ 214us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:11:25.187151 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:11:25.188042 (+ 891us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:11:25.189116 (+ 1074us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:11:25.190403 (+ 1287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:11:25.190972 (+ 569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:11:25.191065 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:11:25.195340 (+ 4275us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:11:25.195351 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:11:25.195398 (+ 47us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:11:25.195448 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:11:25.198319 (+ 2871us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:11:25.198965 (+ 646us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:11:25.198972 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:11:25.198975 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:11:25.199096 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:11:25.199492 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:11:25.199497 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:11:25.199499 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:11:25.199859 (+ 360us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:11:25.200165 (+ 306us) server_negotiation.cc:1035] Waiting for connection context
0928 02:11:25.200479 (+ 314us) server_negotiation.cc:299] Negotiation successful
0928 02:11:25.200729 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":446,"thread_start_us":323,"threads_started":1}
I20260928 02:11:25.202512 13044 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 02:11:25.287267 7085 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTlaUuc/build/asan/bin/kudu with pid 12921
I20260928 02:11:25.304239 7085 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskTlaUuc/build/asan/bin/kudu pid 13055
Sep 28 02:11:25 dist-test-slave-86gv krb5kdc[12681](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561485, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.235.67@KRBTEST.COM
I20260928 02:11:25.320323 13207 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:11:25.305335 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58476 (local address 127.6.235.67:46789)
0928 02:11:25.305980 (+ 645us) server_negotiation.cc:206] Beginning negotiation
0928 02:11:25.305987 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 02:11:25.306015 (+ 28us) server_negotiation.cc:407] Connection header received
0928 02:11:25.306079 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:11:25.306085 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:11:25.306349 (+ 264us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:11:25.306544 (+ 195us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:11:25.307631 (+ 1087us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:11:25.308648 (+ 1017us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:11:25.310090 (+ 1442us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:11:25.310675 (+ 585us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:11:25.310826 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:11:25.314959 (+ 4133us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:11:25.314974 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:11:25.314990 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:11:25.315028 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:11:25.317883 (+ 2855us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:11:25.318545 (+ 662us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:11:25.318552 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:11:25.318555 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:11:25.318673 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:11:25.318996 (+ 323us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:11:25.318999 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:11:25.319001 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:11:25.319342 (+ 341us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:11:25.319560 (+ 218us) server_negotiation.cc:1035] Waiting for connection context
0928 02:11:25.319812 (+ 252us) server_negotiation.cc:299] Negotiation successful
0928 02:11:25.320008 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":500,"thread_start_us":341,"threads_started":1}
I20260928 02:11:25.321919 13179 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 02:11:25.406469 7085 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTlaUuc/build/asan/bin/kudu with pid 13055
I20260928 02:11:25.423183 7085 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskTlaUuc/build/asan/bin/kudu pid 12694
I20260928 02:11:25.424458 12756 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 02:11:25.528136 7085 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTlaUuc/build/asan/bin/kudu with pid 12694
2026-09-28T02:11:25Z chronyd exiting
[ OK ] SecurityITest.TestCorruptKerberosCC (4077 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipalMultipleMaster
Loading random data
Initializing database '/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790561432251112-7085-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 02:11:25 dist-test-slave-86gv krb5kdc[13212](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 02:11:25 dist-test-slave-86gv krb5kdc[13212](info): set up 2 sockets
Sep 28 02:11:25 dist-test-slave-86gv krb5kdc[13212](info): commencing operation
krb5kdc: starting...
W20260928 02:11:26.648288 7085 mini_kdc.cc:121] Time spent starting KDC: real 1.077s user 0.001s sys 0.021s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790561432251112-7085-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790561432251112-7085-0/krb5kdc/test-user.keytab.
Sep 28 02:11:26 dist-test-slave-86gv krb5kdc[13212](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561486, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T02:11:26Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T02:11:26Z Disabled control of system clock
WARNING: no policy specified for oryx/127.6.235.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.6.235.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.6.235.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790561432251112-7085-0/krb5kdc/oryx_127.6.235.126.keytab.
Entry for principal oryx/127.6.235.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790561432251112-7085-0/krb5kdc/oryx_127.6.235.126.keytab.
I20260928 02:11:26.890331 7085 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTlaUuc/build/asan/bin/kudu
/tmp/dist-test-taskTlaUuc/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790561432251112-7085-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790561432251112-7085-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790561432251112-7085-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790561432251112-7085-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.235.126:34853
--webserver_interface=127.6.235.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790561432251112-7085-0/krb5kdc/oryx_127.6.235.126.keytab
--principal=oryx/127.6.235.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.235.84:44519
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.235.126:34853
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790561432251112-7085-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790561432251112-7085-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790561432251112-7085-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:11:27.234017 13225 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:11:27.234357 13225 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:11:27.234462 13225 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:11:27.242276 13225 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 02:11:27.242414 13225 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:11:27.242470 13225 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:11:27.242533 13225 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 02:11:27.242583 13225 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 02:11:27.253100 13225 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.235.84:44519
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790561432251112-7085-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790561432251112-7085-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.235.126:34853
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790561432251112-7085-0/krb5kdc/oryx_127.6.235.126.keytab
--principal=oryx/127.6.235.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.235.126:34853
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790561432251112-7085-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.235.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790561432251112-7085-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 00:51:59 UTC on ccycloud.quasar-axgkwn.root.comops.site
build id 4676464
ASAN enabled
I20260928 02:11:27.254534 13225 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:11:27.255890 13225 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:11:27.269384 13230 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:11:27.269452 13233 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:11:27.269428 13231 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:11:27.270164 13225 server_base.cc:1034] running on GCE node
I20260928 02:11:27.271085 13225 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:11:27.272723 13225 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:11:27.273988 13225 hybrid_clock.cc:648] HybridClock initialized: now 1790561487273941 us; error 59 us; skew 500 ppm
I20260928 02:11:27.274423 13225 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 02:11:27 dist-test-slave-86gv krb5kdc[13212](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561487, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.235.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:11:27.284353 13225 init.cc:377] Logged in from keytab as oryx/127.6.235.126@KRBTEST.COM (short username oryx)
I20260928 02:11:27.286309 13225 webserver.cc:460] Webserver started at http://127.6.235.126:39777/ using document root <none> and password file <none>
I20260928 02:11:27.287170 13225 fs_manager.cc:359] Metadata directory not provided
I20260928 02:11:27.287326 13225 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:11:27.287681 13225 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:11:27.290946 13225 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790561432251112-7085-0/minicluster-data/master-0/data/instance:
uuid: "11adc61886584956adfdac31047373c1"
format_stamp: "Formatted at 2026-09-28 02:11:27 on dist-test-slave-86gv"
server_key: "38dcbc8971ec10abd15c632f49a6d1e5"
server_key_iv: "c3d25a11def60ff0fefeb059449d4c44"
server_key_version: "encryptionkey@0"
I20260928 02:11:27.291795 13225 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790561432251112-7085-0/minicluster-data/master-0/wal/instance:
uuid: "11adc61886584956adfdac31047373c1"
format_stamp: "Formatted at 2026-09-28 02:11:27 on dist-test-slave-86gv"
server_key: "38dcbc8971ec10abd15c632f49a6d1e5"
server_key_iv: "c3d25a11def60ff0fefeb059449d4c44"
server_key_version: "encryptionkey@0"
I20260928 02:11:27.297649 13225 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260928 02:11:27.302346 13240 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:11:27.303996 13225 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.003s
I20260928 02:11:27.304280 13225 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790561432251112-7085-0/minicluster-data/master-0/data,/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790561432251112-7085-0/minicluster-data/master-0/wal
uuid: "11adc61886584956adfdac31047373c1"
format_stamp: "Formatted at 2026-09-28 02:11:27 on dist-test-slave-86gv"
server_key: "38dcbc8971ec10abd15c632f49a6d1e5"
server_key_iv: "c3d25a11def60ff0fefeb059449d4c44"
server_key_version: "encryptionkey@0"
I20260928 02:11:27.304479 13225 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790561432251112-7085-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790561432251112-7085-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790561432251112-7085-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:11:27.341204 13225 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:11:27.346015 13225 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:11:27.346343 13225 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:11:27.365435 13225 rpc_server.cc:307] RPC server started. Bound to: 127.6.235.126:34853
I20260928 02:11:27.365469 13291 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.235.126:34853 every 8 connection(s)
I20260928 02:11:27.366901 13225 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790561432251112-7085-0/minicluster-data/master-0/data/info.pb
I20260928 02:11:27.371151 13292 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:11:27.374311 7085 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTlaUuc/build/asan/bin/kudu as pid 13225
I20260928 02:11:27.374557 7085 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790561432251112-7085-0/minicluster-data/master-0/wal/instance
I20260928 02:11:27.375162 7085 external_mini_cluster.cc:1428] Setting key 12f696a35bc63a81fb764905638cfbcf
I20260928 02:11:27.385041 13292 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 11adc61886584956adfdac31047373c1: Bootstrap starting.
I20260928 02:11:27.388841 13292 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 11adc61886584956adfdac31047373c1: Neither blocks nor log segments found. Creating new log.
I20260928 02:11:27.389961 13292 log.cc:824] T 00000000000000000000000000000000 P 11adc61886584956adfdac31047373c1: Log is configured to *not* fsync() on all Append() calls
Sep 28 02:11:27 dist-test-slave-86gv krb5kdc[13212](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561486, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.6.235.126@KRBTEST.COM
I20260928 02:11:27.393399 13292 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 11adc61886584956adfdac31047373c1: No bootstrap required, opened a new log
I20260928 02:11:27.400590 13295 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:11:27.377452 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42062 (local address 127.6.235.126:34853)
0928 02:11:27.378308 (+ 856us) server_negotiation.cc:206] Beginning negotiation
0928 02:11:27.378325 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0928 02:11:27.378391 (+ 66us) server_negotiation.cc:407] Connection header received
0928 02:11:27.380620 (+ 2229us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:11:27.380656 (+ 36us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:11:27.381339 (+ 683us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:11:27.381863 (+ 524us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:11:27.383323 (+ 1460us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:11:27.385312 (+ 1989us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:11:27.386628 (+ 1316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:11:27.387685 (+ 1057us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:11:27.387892 (+ 207us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:11:27.392332 (+ 4440us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:11:27.392339 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:11:27.392354 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:11:27.392398 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:11:27.395859 (+ 3461us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:11:27.396791 (+ 932us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:11:27.396799 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:11:27.396806 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:11:27.397047 (+ 241us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:11:27.397489 (+ 442us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:11:27.397492 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:11:27.397494 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:11:27.398098 (+ 604us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:11:27.398367 (+ 269us) server_negotiation.cc:1035] Waiting for connection context
0928 02:11:27.398962 (+ 595us) server_negotiation.cc:299] Negotiation successful
0928 02:11:27.399452 (+ 490us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":600,"thread_start_us":365,"threads_started":1}
I20260928 02:11:27.403004 13292 raft_consensus.cc:348] T 00000000000000000000000000000000 P 11adc61886584956adfdac31047373c1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "11adc61886584956adfdac31047373c1" member_type: VOTER last_known_addr { host: "127.6.235.126" port: 34853 } }
I20260928 02:11:27.403419 13292 raft_consensus.cc:374] T 00000000000000000000000000000000 P 11adc61886584956adfdac31047373c1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:11:27.403541 13292 raft_consensus.cc:729] T 00000000000000000000000000000000 P 11adc61886584956adfdac31047373c1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 11adc61886584956adfdac31047373c1, State: Initialized, Role: FOLLOWER
I20260928 02:11:27.404247 13292 consensus_queue.cc:260] T 00000000000000000000000000000000 P 11adc61886584956adfdac31047373c1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "11adc61886584956adfdac31047373c1" member_type: VOTER last_known_addr { host: "127.6.235.126" port: 34853 } }
I20260928 02:11:27.404507 13292 raft_consensus.cc:388] T 00000000000000000000000000000000 P 11adc61886584956adfdac31047373c1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 02:11:27.404649 13292 raft_consensus.cc:482] T 00000000000000000000000000000000 P 11adc61886584956adfdac31047373c1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 02:11:27.404836 13292 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 11adc61886584956adfdac31047373c1 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:11:27.407229 13292 raft_consensus.cc:504] T 00000000000000000000000000000000 P 11adc61886584956adfdac31047373c1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "11adc61886584956adfdac31047373c1" member_type: VOTER last_known_addr { host: "127.6.235.126" port: 34853 } }
I20260928 02:11:27.407793 13292 leader_election.cc:302] T 00000000000000000000000000000000 P 11adc61886584956adfdac31047373c1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 11adc61886584956adfdac31047373c1; no voters:
I20260928 02:11:27.408445 13292 leader_election.cc:288] T 00000000000000000000000000000000 P 11adc61886584956adfdac31047373c1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 02:11:27.408564 13297 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 11adc61886584956adfdac31047373c1 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:11:27.409492 13297 raft_consensus.cc:686] T 00000000000000000000000000000000 P 11adc61886584956adfdac31047373c1 [term 1 LEADER]: Becoming Leader. State: Replica: 11adc61886584956adfdac31047373c1, State: Running, Role: LEADER
I20260928 02:11:27.410060 13297 consensus_queue.cc:237] T 00000000000000000000000000000000 P 11adc61886584956adfdac31047373c1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "11adc61886584956adfdac31047373c1" member_type: VOTER last_known_addr { host: "127.6.235.126" port: 34853 } }
I20260928 02:11:27.410879 13292 sys_catalog.cc:564] T 00000000000000000000000000000000 P 11adc61886584956adfdac31047373c1 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 02:11:27.413017 13299 sys_catalog.cc:455] T 00000000000000000000000000000000 P 11adc61886584956adfdac31047373c1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 11adc61886584956adfdac31047373c1. Latest consensus state: current_term: 1 leader_uuid: "11adc61886584956adfdac31047373c1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "11adc61886584956adfdac31047373c1" member_type: VOTER last_known_addr { host: "127.6.235.126" port: 34853 } } }
I20260928 02:11:27.413442 13299 sys_catalog.cc:458] T 00000000000000000000000000000000 P 11adc61886584956adfdac31047373c1 [sys.catalog]: This master's current role is: LEADER
I20260928 02:11:27.414474 13306 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 02:11:27.414392 13298 sys_catalog.cc:455] T 00000000000000000000000000000000 P 11adc61886584956adfdac31047373c1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "11adc61886584956adfdac31047373c1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "11adc61886584956adfdac31047373c1" member_type: VOTER last_known_addr { host: "127.6.235.126" port: 34853 } } }
I20260928 02:11:27.415347 13298 sys_catalog.cc:458] T 00000000000000000000000000000000 P 11adc61886584956adfdac31047373c1 [sys.catalog]: This master's current role is: LEADER
I20260928 02:11:27.439620 13306 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 02:11:27.449368 13306 catalog_manager.cc:1347] Generated new cluster ID: 33ac180e627642b2a4850f9fab0c8be0
I20260928 02:11:27.449502 13306 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 02:11:27.468014 13306 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 02:11:27.470134 13306 catalog_manager.cc:1504] Loading token signing keys...
I20260928 02:11:27.476271 13306 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 11adc61886584956adfdac31047373c1: Generated new TSK 0
I20260928 02:11:27.477275 13306 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.6.235.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.6.235.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.6.235.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790561432251112-7085-0/krb5kdc/oryx_127.6.235.65.keytab.
Entry for principal oryx/127.6.235.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790561432251112-7085-0/krb5kdc/oryx_127.6.235.65.keytab.
I20260928 02:11:27.545567 7085 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTlaUuc/build/asan/bin/kudu
/tmp/dist-test-taskTlaUuc/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790561432251112-7085-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790561432251112-7085-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790561432251112-7085-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790561432251112-7085-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.235.65:0
--local_ip_for_outbound_sockets=127.6.235.65
--webserver_interface=127.6.235.65
--webserver_port=0
--tserver_master_addrs=127.6.235.126:34853
--keytab_file=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790561432251112-7085-0/krb5kdc/oryx_127.6.235.65.keytab
--principal=oryx/127.6.235.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.235.84:44519
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790561432251112-7085-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790561432251112-7085-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790561432251112-7085-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:11:27.898625 13318 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:11:27.898937 13318 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:11:27.899031 13318 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:11:27.906679 13318 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:11:27.906874 13318 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:11:27.906996 13318 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.235.65
I20260928 02:11:27.918323 13318 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.235.84:44519
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790561432251112-7085-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790561432251112-7085-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790561432251112-7085-0/krb5kdc/oryx_127.6.235.65.keytab
--principal=oryx/127.6.235.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.235.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790561432251112-7085-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.6.235.65
--webserver_port=0
--tserver_master_addrs=127.6.235.126:34853
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.235.65
--log_dir=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790561432251112-7085-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 00:51:59 UTC on ccycloud.quasar-axgkwn.root.comops.site
build id 4676464
ASAN enabled
I20260928 02:11:27.919761 13318 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:11:27.921403 13318 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:11:27.936177 13323 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:11:27.936242 13326 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:11:27.936221 13324 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:11:27.937350 13318 server_base.cc:1034] running on GCE node
I20260928 02:11:27.938035 13318 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:11:27.939081 13318 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:11:27.940384 13318 hybrid_clock.cc:648] HybridClock initialized: now 1790561487940304 us; error 40 us; skew 500 ppm
I20260928 02:11:27.940821 13318 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 02:11:27 dist-test-slave-86gv krb5kdc[13212](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561487, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.235.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:11:27.944568 13318 init.cc:377] Logged in from keytab as oryx/127.6.235.65@KRBTEST.COM (short username oryx)
I20260928 02:11:27.946308 13318 webserver.cc:460] Webserver started at http://127.6.235.65:34129/ using document root <none> and password file <none>
I20260928 02:11:27.947068 13318 fs_manager.cc:359] Metadata directory not provided
I20260928 02:11:27.947175 13318 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:11:27.947465 13318 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:11:27.950891 13318 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790561432251112-7085-0/minicluster-data/ts-0/data/instance:
uuid: "db7477740b844e9c8e8d14f8724f78d3"
format_stamp: "Formatted at 2026-09-28 02:11:27 on dist-test-slave-86gv"
server_key: "0f1b47083a67517a1f97fd57748b86f9"
server_key_iv: "8f217dfca3ef6ce3ddde88656bca0f10"
server_key_version: "encryptionkey@0"
I20260928 02:11:27.951709 13318 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790561432251112-7085-0/minicluster-data/ts-0/wal/instance:
uuid: "db7477740b844e9c8e8d14f8724f78d3"
format_stamp: "Formatted at 2026-09-28 02:11:27 on dist-test-slave-86gv"
server_key: "0f1b47083a67517a1f97fd57748b86f9"
server_key_iv: "8f217dfca3ef6ce3ddde88656bca0f10"
server_key_version: "encryptionkey@0"
I20260928 02:11:27.957577 13318 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.005s
I20260928 02:11:27.961457 13333 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:11:27.962986 13318 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260928 02:11:27.963163 13318 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790561432251112-7085-0/minicluster-data/ts-0/data,/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790561432251112-7085-0/minicluster-data/ts-0/wal
uuid: "db7477740b844e9c8e8d14f8724f78d3"
format_stamp: "Formatted at 2026-09-28 02:11:27 on dist-test-slave-86gv"
server_key: "0f1b47083a67517a1f97fd57748b86f9"
server_key_iv: "8f217dfca3ef6ce3ddde88656bca0f10"
server_key_version: "encryptionkey@0"
I20260928 02:11:27.963361 13318 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790561432251112-7085-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790561432251112-7085-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790561432251112-7085-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:11:27.981343 13318 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:11:27.986186 13318 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:11:27.986505 13318 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:11:27.987615 13318 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:11:27.989169 13318 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:11:27.989244 13318 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:11:27.989413 13318 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:11:27.989481 13318 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:11:28.024977 13318 rpc_server.cc:307] RPC server started. Bound to: 127.6.235.65:42113
I20260928 02:11:28.024982 13445 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.235.65:42113 every 8 connection(s)
I20260928 02:11:28.026466 13318 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790561432251112-7085-0/minicluster-data/ts-0/data/info.pb
I20260928 02:11:28.029645 7085 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTlaUuc/build/asan/bin/kudu as pid 13318
I20260928 02:11:28.029892 7085 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790561432251112-7085-0/minicluster-data/ts-0/wal/instance
I20260928 02:11:28.030619 7085 external_mini_cluster.cc:1428] Setting key 25316d22104d7b5035bdd77d5ea1acd3
Sep 28 02:11:28 dist-test-slave-86gv krb5kdc[13212](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561487, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.235.65@KRBTEST.COM for oryx/127.6.235.126@KRBTEST.COM
I20260928 02:11:28.052296 13448 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:11:28.029434 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.235.65:53197 (local address 127.6.235.126:34853)
0928 02:11:28.029952 (+ 518us) server_negotiation.cc:206] Beginning negotiation
0928 02:11:28.029961 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 02:11:28.031207 (+ 1246us) server_negotiation.cc:407] Connection header received
0928 02:11:28.033220 (+ 2013us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:11:28.033225 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:11:28.033290 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:11:28.033387 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:11:28.035787 (+ 2400us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:11:28.036684 (+ 897us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:11:28.039464 (+ 2780us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:11:28.039926 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:11:28.040155 (+ 229us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:11:28.045022 (+ 4867us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:11:28.045047 (+ 25us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:11:28.045051 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:11:28.045154 (+ 103us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:11:28.047896 (+ 2742us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:11:28.049415 (+ 1519us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:11:28.049420 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:11:28.049422 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:11:28.049554 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:11:28.050181 (+ 627us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:11:28.050186 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:11:28.050189 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:11:28.050505 (+ 316us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:11:28.050640 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0928 02:11:28.051702 (+ 1062us) server_negotiation.cc:299] Negotiation successful
0928 02:11:28.051913 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":370,"thread_start_us":230,"threads_started":1}
I20260928 02:11:28.054005 13449 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:11:28.030113 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.235.126:34853 (local address 127.6.235.65:53197)
0928 02:11:28.030998 (+ 885us) negotiation.cc:107] Waiting for socket to connect
0928 02:11:28.031062 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0928 02:11:28.032850 (+ 1788us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:11:28.033713 (+ 863us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:11:28.033728 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:11:28.034598 (+ 870us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:11:28.035542 (+ 944us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:11:28.035572 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:11:28.036883 (+ 1311us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:11:28.036921 (+ 38us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:11:28.039255 (+ 2334us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:11:28.039276 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:11:28.040213 (+ 937us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:11:28.040222 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:11:28.040579 (+ 357us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:11:28.041567 (+ 988us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:11:28.041602 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:11:28.044671 (+ 3069us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:11:28.048179 (+ 3508us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:11:28.048194 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:11:28.048218 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:11:28.048956 (+ 738us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:11:28.049704 (+ 748us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:11:28.049711 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:11:28.049718 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:11:28.050004 (+ 286us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:11:28.050657 (+ 653us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:11:28.050669 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:11:28.051230 (+ 561us) client_negotiation.cc:769] Sending connection context
0928 02:11:28.051580 (+ 350us) client_negotiation.cc:240] Negotiation successful
0928 02:11:28.052267 (+ 687us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":574,"thread_start_us":330,"threads_started":1}
WARNING: no policy specified for oryx/127.6.235.66@KRBTEST.COM; defaulting to no policy
I20260928 02:11:28.056156 13446 heartbeater.cc:347] Connected to a master server at 127.6.235.126:34853
I20260928 02:11:28.056540 13446 heartbeater.cc:464] Registering TS with master...
I20260928 02:11:28.057487 13446 heartbeater.cc:511] Master 127.6.235.126:34853 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.6.235.66@KRBTEST.COM" created.
I20260928 02:11:28.060273 13257 ts_manager.cc:194] Registered new tserver with Master: db7477740b844e9c8e8d14f8724f78d3 (127.6.235.65:42113)
I20260928 02:11:28.061954 13257 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.6.235.65@KRBTEST.COM'} at 127.6.235.65:53197
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.6.235.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790561432251112-7085-0/krb5kdc/oryx_127.6.235.66.keytab.
Entry for principal oryx/127.6.235.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790561432251112-7085-0/krb5kdc/oryx_127.6.235.66.keytab.
I20260928 02:11:28.090243 7085 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTlaUuc/build/asan/bin/kudu
/tmp/dist-test-taskTlaUuc/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790561432251112-7085-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790561432251112-7085-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790561432251112-7085-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790561432251112-7085-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.235.66:0
--local_ip_for_outbound_sockets=127.6.235.66
--webserver_interface=127.6.235.66
--webserver_port=0
--tserver_master_addrs=127.6.235.126:34853
--keytab_file=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790561432251112-7085-0/krb5kdc/oryx_127.6.235.66.keytab
--principal=oryx/127.6.235.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.235.84:44519
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790561432251112-7085-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790561432251112-7085-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790561432251112-7085-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:11:28.447199 13452 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:11:28.447556 13452 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:11:28.447726 13452 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:11:28.455224 13452 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:11:28.455403 13452 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:11:28.455600 13452 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.235.66
I20260928 02:11:28.466168 13452 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.235.84:44519
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790561432251112-7085-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790561432251112-7085-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790561432251112-7085-0/krb5kdc/oryx_127.6.235.66.keytab
--principal=oryx/127.6.235.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.235.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790561432251112-7085-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.6.235.66
--webserver_port=0
--tserver_master_addrs=127.6.235.126:34853
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.235.66
--log_dir=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790561432251112-7085-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 00:51:59 UTC on ccycloud.quasar-axgkwn.root.comops.site
build id 4676464
ASAN enabled
I20260928 02:11:28.467656 13452 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:11:28.469241 13452 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:11:28.484306 13460 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:11:28.484412 13457 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:11:28.484416 13458 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:11:28.486313 13452 server_base.cc:1034] running on GCE node
I20260928 02:11:28.487056 13452 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:11:28.488269 13452 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:11:28.489565 13452 hybrid_clock.cc:648] HybridClock initialized: now 1790561488489544 us; error 113 us; skew 500 ppm
I20260928 02:11:28.490006 13452 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 02:11:28 dist-test-slave-86gv krb5kdc[13212](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561488, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.235.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:11:28.493654 13452 init.cc:377] Logged in from keytab as oryx/127.6.235.66@KRBTEST.COM (short username oryx)
I20260928 02:11:28.495222 13452 webserver.cc:460] Webserver started at http://127.6.235.66:33027/ using document root <none> and password file <none>
I20260928 02:11:28.495921 13452 fs_manager.cc:359] Metadata directory not provided
I20260928 02:11:28.496022 13452 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:11:28.496284 13452 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:11:28.499329 13452 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790561432251112-7085-0/minicluster-data/ts-1/data/instance:
uuid: "9f8e4ca0aeac43aa8ab6f8ab9a4d6fe9"
format_stamp: "Formatted at 2026-09-28 02:11:28 on dist-test-slave-86gv"
server_key: "b185fb7050e8c310d8982c65add9c482"
server_key_iv: "fd730cf34d10e397e130f0dc2e66479c"
server_key_version: "encryptionkey@0"
I20260928 02:11:28.500097 13452 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790561432251112-7085-0/minicluster-data/ts-1/wal/instance:
uuid: "9f8e4ca0aeac43aa8ab6f8ab9a4d6fe9"
format_stamp: "Formatted at 2026-09-28 02:11:28 on dist-test-slave-86gv"
server_key: "b185fb7050e8c310d8982c65add9c482"
server_key_iv: "fd730cf34d10e397e130f0dc2e66479c"
server_key_version: "encryptionkey@0"
I20260928 02:11:28.505707 13452 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260928 02:11:28.509604 13467 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:11:28.511323 13452 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.004s
I20260928 02:11:28.511503 13452 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790561432251112-7085-0/minicluster-data/ts-1/data,/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790561432251112-7085-0/minicluster-data/ts-1/wal
uuid: "9f8e4ca0aeac43aa8ab6f8ab9a4d6fe9"
format_stamp: "Formatted at 2026-09-28 02:11:28 on dist-test-slave-86gv"
server_key: "b185fb7050e8c310d8982c65add9c482"
server_key_iv: "fd730cf34d10e397e130f0dc2e66479c"
server_key_version: "encryptionkey@0"
I20260928 02:11:28.511740 13452 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790561432251112-7085-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790561432251112-7085-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790561432251112-7085-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:11:28.550117 13452 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:11:28.555294 13452 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:11:28.555624 13452 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:11:28.556761 13452 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:11:28.558238 13452 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:11:28.558318 13452 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:11:28.558447 13452 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:11:28.558496 13452 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:11:28.592908 13452 rpc_server.cc:307] RPC server started. Bound to: 127.6.235.66:42855
I20260928 02:11:28.593016 13579 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.235.66:42855 every 8 connection(s)
I20260928 02:11:28.594318 13452 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790561432251112-7085-0/minicluster-data/ts-1/data/info.pb
I20260928 02:11:28.594765 7085 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTlaUuc/build/asan/bin/kudu as pid 13452
I20260928 02:11:28.594995 7085 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790561432251112-7085-0/minicluster-data/ts-1/wal/instance
I20260928 02:11:28.595585 7085 external_mini_cluster.cc:1428] Setting key 9bafd15a7ac2e93af2b2064f87f3eea8
Sep 28 02:11:28 dist-test-slave-86gv krb5kdc[13212](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561488, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.235.66@KRBTEST.COM for oryx/127.6.235.126@KRBTEST.COM
I20260928 02:11:28.617519 13582 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:11:28.597046 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.235.66:50673 (local address 127.6.235.126:34853)
0928 02:11:28.597528 (+ 482us) server_negotiation.cc:206] Beginning negotiation
0928 02:11:28.597535 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 02:11:28.598701 (+ 1166us) server_negotiation.cc:407] Connection header received
0928 02:11:28.600548 (+ 1847us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:11:28.600553 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:11:28.600642 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:11:28.600773 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:11:28.602533 (+ 1760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:11:28.603263 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:11:28.605553 (+ 2290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:11:28.606045 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:11:28.606165 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:11:28.610678 (+ 4513us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:11:28.610684 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:11:28.610694 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:11:28.610730 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:11:28.613869 (+ 3139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:11:28.614875 (+ 1006us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:11:28.614879 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:11:28.614882 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:11:28.615007 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:11:28.615547 (+ 540us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:11:28.615552 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:11:28.615555 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:11:28.615844 (+ 289us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:11:28.616013 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0928 02:11:28.616797 (+ 784us) server_negotiation.cc:299] Negotiation successful
0928 02:11:28.617059 (+ 262us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":358,"thread_start_us":259,"threads_started":1}
WARNING: no policy specified for oryx/127.6.235.67@KRBTEST.COM; defaulting to no policy
I20260928 02:11:28.618551 13583 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:11:28.597734 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.235.126:34853 (local address 127.6.235.66:50673)
0928 02:11:28.598542 (+ 808us) negotiation.cc:107] Waiting for socket to connect
0928 02:11:28.598586 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0928 02:11:28.600266 (+ 1680us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:11:28.601050 (+ 784us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:11:28.601059 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:11:28.601758 (+ 699us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:11:28.602350 (+ 592us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:11:28.602369 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:11:28.603423 (+ 1054us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:11:28.603435 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:11:28.605330 (+ 1895us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:11:28.605350 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:11:28.606299 (+ 949us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:11:28.606308 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:11:28.606652 (+ 344us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:11:28.607477 (+ 825us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:11:28.607513 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:11:28.610405 (+ 2892us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:11:28.614038 (+ 3633us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:11:28.614048 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:11:28.614068 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:11:28.614734 (+ 666us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:11:28.615214 (+ 480us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:11:28.615219 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:11:28.615223 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:11:28.615426 (+ 203us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:11:28.615987 (+ 561us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:11:28.615995 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:11:28.616419 (+ 424us) client_negotiation.cc:769] Sending connection context
0928 02:11:28.616706 (+ 287us) client_negotiation.cc:240] Negotiation successful
0928 02:11:28.617304 (+ 598us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":576,"thread_start_us":373,"threads_started":1}
I20260928 02:11:28.620108 13580 heartbeater.cc:347] Connected to a master server at 127.6.235.126:34853
I20260928 02:11:28.620493 13580 heartbeater.cc:464] Registering TS with master...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.6.235.67@KRBTEST.COM" created.
I20260928 02:11:28.621423 13580 heartbeater.cc:511] Master 127.6.235.126:34853 requested a full tablet report, sending...
I20260928 02:11:28.623203 13257 ts_manager.cc:194] Registered new tserver with Master: 9f8e4ca0aeac43aa8ab6f8ab9a4d6fe9 (127.6.235.66:42855)
I20260928 02:11:28.624432 13257 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.6.235.66@KRBTEST.COM'} at 127.6.235.66:50673
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.6.235.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790561432251112-7085-0/krb5kdc/oryx_127.6.235.67.keytab.
Entry for principal oryx/127.6.235.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790561432251112-7085-0/krb5kdc/oryx_127.6.235.67.keytab.
I20260928 02:11:28.656863 7085 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTlaUuc/build/asan/bin/kudu
/tmp/dist-test-taskTlaUuc/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790561432251112-7085-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790561432251112-7085-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790561432251112-7085-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790561432251112-7085-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.235.67:0
--local_ip_for_outbound_sockets=127.6.235.67
--webserver_interface=127.6.235.67
--webserver_port=0
--tserver_master_addrs=127.6.235.126:34853
--keytab_file=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790561432251112-7085-0/krb5kdc/oryx_127.6.235.67.keytab
--principal=oryx/127.6.235.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.235.84:44519
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790561432251112-7085-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790561432251112-7085-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790561432251112-7085-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:11:29.004647 13586 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:11:29.005025 13586 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:11:29.005213 13586 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:11:29.013435 13586 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:11:29.013598 13586 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:11:29.013810 13586 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.235.67
I20260928 02:11:29.024317 13586 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.235.84:44519
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790561432251112-7085-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790561432251112-7085-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790561432251112-7085-0/krb5kdc/oryx_127.6.235.67.keytab
--principal=oryx/127.6.235.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.235.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790561432251112-7085-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.6.235.67
--webserver_port=0
--tserver_master_addrs=127.6.235.126:34853
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.235.67
--log_dir=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790561432251112-7085-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 00:51:59 UTC on ccycloud.quasar-axgkwn.root.comops.site
build id 4676464
ASAN enabled
I20260928 02:11:29.025807 13586 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:11:29.027177 13586 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:11:29.040761 13594 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:11:29.040762 13591 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:11:29.040925 13592 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:11:29.041751 13586 server_base.cc:1034] running on GCE node
I20260928 02:11:29.042379 13586 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:11:29.043390 13586 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:11:29.044601 13586 hybrid_clock.cc:648] HybridClock initialized: now 1790561489044559 us; error 33 us; skew 500 ppm
I20260928 02:11:29.045029 13586 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 02:11:29 dist-test-slave-86gv krb5kdc[13212](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561489, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.235.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:11:29.048286 13586 init.cc:377] Logged in from keytab as oryx/127.6.235.67@KRBTEST.COM (short username oryx)
I20260928 02:11:29.049961 13586 webserver.cc:460] Webserver started at http://127.6.235.67:43293/ using document root <none> and password file <none>
I20260928 02:11:29.050704 13586 fs_manager.cc:359] Metadata directory not provided
I20260928 02:11:29.050832 13586 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:11:29.051153 13586 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:11:29.054260 13586 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790561432251112-7085-0/minicluster-data/ts-2/data/instance:
uuid: "9dae072a16c44f1a8e4fe720ecff93cc"
format_stamp: "Formatted at 2026-09-28 02:11:29 on dist-test-slave-86gv"
server_key: "4e53bee3367eb30daf885b3a1365e5f8"
server_key_iv: "0d61598ee067e647ab16c18947542468"
server_key_version: "encryptionkey@0"
I20260928 02:11:29.055135 13586 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790561432251112-7085-0/minicluster-data/ts-2/wal/instance:
uuid: "9dae072a16c44f1a8e4fe720ecff93cc"
format_stamp: "Formatted at 2026-09-28 02:11:29 on dist-test-slave-86gv"
server_key: "4e53bee3367eb30daf885b3a1365e5f8"
server_key_iv: "0d61598ee067e647ab16c18947542468"
server_key_version: "encryptionkey@0"
I20260928 02:11:29.060633 13586 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260928 02:11:29.064499 13601 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:11:29.065614 13446 heartbeater.cc:503] Master 127.6.235.126:34853 was elected leader, sending a full tablet report...
I20260928 02:11:29.066282 13586 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 02:11:29.066448 13586 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790561432251112-7085-0/minicluster-data/ts-2/data,/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790561432251112-7085-0/minicluster-data/ts-2/wal
uuid: "9dae072a16c44f1a8e4fe720ecff93cc"
format_stamp: "Formatted at 2026-09-28 02:11:29 on dist-test-slave-86gv"
server_key: "4e53bee3367eb30daf885b3a1365e5f8"
server_key_iv: "0d61598ee067e647ab16c18947542468"
server_key_version: "encryptionkey@0"
I20260928 02:11:29.066630 13586 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790561432251112-7085-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790561432251112-7085-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790561432251112-7085-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:11:29.086621 13586 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:11:29.091264 13586 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:11:29.091602 13586 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:11:29.092686 13586 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:11:29.094269 13586 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:11:29.094342 13586 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:11:29.094497 13586 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:11:29.094576 13586 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:11:29.129900 13586 rpc_server.cc:307] RPC server started. Bound to: 127.6.235.67:40303
I20260928 02:11:29.129992 13713 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.235.67:40303 every 8 connection(s)
I20260928 02:11:29.131213 13586 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790561432251112-7085-0/minicluster-data/ts-2/data/info.pb
I20260928 02:11:29.140288 7085 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTlaUuc/build/asan/bin/kudu as pid 13586
I20260928 02:11:29.140491 7085 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790561432251112-7085-0/minicluster-data/ts-2/wal/instance
I20260928 02:11:29.140966 7085 external_mini_cluster.cc:1428] Setting key 647994c91c54992785a27110394fcfd2
Sep 28 02:11:29 dist-test-slave-86gv krb5kdc[13212](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561489, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.235.67@KRBTEST.COM for oryx/127.6.235.126@KRBTEST.COM
I20260928 02:11:29.151615 13716 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:11:29.134009 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.235.67:49265 (local address 127.6.235.126:34853)
0928 02:11:29.134492 (+ 483us) server_negotiation.cc:206] Beginning negotiation
0928 02:11:29.134507 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0928 02:11:29.135562 (+ 1055us) server_negotiation.cc:407] Connection header received
0928 02:11:29.137358 (+ 1796us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:11:29.137363 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:11:29.137443 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:11:29.137542 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:11:29.139169 (+ 1627us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:11:29.140030 (+ 861us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:11:29.141848 (+ 1818us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:11:29.142405 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:11:29.142541 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:11:29.146282 (+ 3741us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:11:29.146287 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:11:29.146291 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:11:29.146324 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:11:29.148764 (+ 2440us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:11:29.149532 (+ 768us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:11:29.149536 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:11:29.149539 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:11:29.149658 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:11:29.150071 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:11:29.150074 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:11:29.150077 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:11:29.150325 (+ 248us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:11:29.150434 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0928 02:11:29.151145 (+ 711us) server_negotiation.cc:299] Negotiation successful
0928 02:11:29.151335 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":358,"thread_start_us":228,"threads_started":1}
I20260928 02:11:29.152598 13717 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:11:29.134669 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.235.126:34853 (local address 127.6.235.67:49265)
0928 02:11:29.135396 (+ 727us) negotiation.cc:107] Waiting for socket to connect
0928 02:11:29.135444 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0928 02:11:29.137106 (+ 1662us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:11:29.137838 (+ 732us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:11:29.137848 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:11:29.138407 (+ 559us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:11:29.139003 (+ 596us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:11:29.139020 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:11:29.140206 (+ 1186us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:11:29.140211 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:11:29.141720 (+ 1509us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:11:29.141732 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:11:29.142620 (+ 888us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:11:29.142625 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:11:29.142909 (+ 284us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:11:29.143659 (+ 750us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:11:29.143693 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:11:29.146072 (+ 2379us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:11:29.148948 (+ 2876us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:11:29.148956 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:11:29.148971 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:11:29.149415 (+ 444us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:11:29.149787 (+ 372us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:11:29.149791 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:11:29.149793 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:11:29.149970 (+ 177us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:11:29.150430 (+ 460us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:11:29.150436 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:11:29.150808 (+ 372us) client_negotiation.cc:769] Sending connection context
0928 02:11:29.151038 (+ 230us) client_negotiation.cc:240] Negotiation successful
0928 02:11:29.151453 (+ 415us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":487,"thread_start_us":300,"threads_started":1}
I20260928 02:11:29.154083 13714 heartbeater.cc:347] Connected to a master server at 127.6.235.126:34853
I20260928 02:11:29.154431 13714 heartbeater.cc:464] Registering TS with master...
I20260928 02:11:29.155340 13714 heartbeater.cc:511] Master 127.6.235.126:34853 requested a full tablet report, sending...
I20260928 02:11:29.157017 13257 ts_manager.cc:194] Registered new tserver with Master: 9dae072a16c44f1a8e4fe720ecff93cc (127.6.235.67:40303)
I20260928 02:11:29.157831 13257 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.6.235.67@KRBTEST.COM'} at 127.6.235.67:49265
I20260928 02:11:29.169538 7085 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.6.235.125@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.6.235.125@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.6.235.125 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790561432251112-7085-0/krb5kdc/oryx_127.6.235.125.keytab.
Entry for principal oryx/127.6.235.125 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790561432251112-7085-0/krb5kdc/oryx_127.6.235.125.keytab.
I20260928 02:11:29.222647 7085 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTlaUuc/build/asan/bin/kudu
/tmp/dist-test-taskTlaUuc/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790561432251112-7085-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790561432251112-7085-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790561432251112-7085-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790561432251112-7085-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.235.125:41279
--webserver_interface=127.6.235.125
--webserver_port=0
--keytab_file=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790561432251112-7085-0/krb5kdc/oryx_127.6.235.125.keytab
--principal=oryx/127.6.235.125
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.235.84:44519
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.235.126:34853,127.6.235.125:41279
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790561432251112-7085-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790561432251112-7085-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790561432251112-7085-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:11:29.576963 13720 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:11:29.577324 13720 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:11:29.577471 13720 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:11:29.585183 13720 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 02:11:29.585320 13720 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:11:29.585404 13720 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:11:29.585486 13720 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 02:11:29.585552 13720 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 02:11:29.595701 13720 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.235.84:44519
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790561432251112-7085-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790561432251112-7085-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.6.235.126:34853,127.6.235.125:41279
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790561432251112-7085-0/krb5kdc/oryx_127.6.235.125.keytab
--principal=oryx/127.6.235.125
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.235.125:41279
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790561432251112-7085-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.6.235.125
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790561432251112-7085-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 00:51:59 UTC on ccycloud.quasar-axgkwn.root.comops.site
build id 4676464
ASAN enabled
W20260928 02:11:29.596114 13720 master_options.cc:55] Only 2 masters are specified by master_addresses_flag ('127.6.235.126:34853,127.6.235.125:41279'), but minimum of 3 are required to tolerate failures of any one master. It is recommended to use at least 3 masters.
I20260928 02:11:29.597256 13720 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:11:29.598553 13720 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:11:29.610400 13725 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:11:29.610445 13726 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:11:29.610452 13728 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:11:29.611282 13720 server_base.cc:1034] running on GCE node
I20260928 02:11:29.612084 13720 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:11:29.613683 13720 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:11:29.614926 13720 hybrid_clock.cc:648] HybridClock initialized: now 1790561489614887 us; error 41 us; skew 500 ppm
I20260928 02:11:29.615299 13720 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 02:11:29 dist-test-slave-86gv krb5kdc[13212](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561489, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.235.125@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:11:29.618512 13720 init.cc:377] Logged in from keytab as oryx/127.6.235.125@KRBTEST.COM (short username oryx)
I20260928 02:11:29.620096 13720 webserver.cc:460] Webserver started at http://127.6.235.125:42699/ using document root <none> and password file <none>
I20260928 02:11:29.620828 13720 fs_manager.cc:359] Metadata directory not provided
I20260928 02:11:29.620985 13720 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:11:29.621265 13720 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:11:29.624131 13720 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790561432251112-7085-0/minicluster-data/master-1/data/instance:
uuid: "bafe67917b9a4b40a60409c753bf4b96"
format_stamp: "Formatted at 2026-09-28 02:11:29 on dist-test-slave-86gv"
server_key: "c5221606d2f99f636c600e995e76e14c"
server_key_iv: "d4eaff2fdba8a601b8abafbe6a9d7389"
server_key_version: "encryptionkey@0"
I20260928 02:11:29.624866 13720 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790561432251112-7085-0/minicluster-data/master-1/wal/instance:
uuid: "bafe67917b9a4b40a60409c753bf4b96"
format_stamp: "Formatted at 2026-09-28 02:11:29 on dist-test-slave-86gv"
server_key: "c5221606d2f99f636c600e995e76e14c"
server_key_iv: "d4eaff2fdba8a601b8abafbe6a9d7389"
server_key_version: "encryptionkey@0"
I20260928 02:11:29.627962 13580 heartbeater.cc:503] Master 127.6.235.126:34853 was elected leader, sending a full tablet report...
I20260928 02:11:29.630542 13720 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.003s
I20260928 02:11:29.634059 13735 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:11:29.635494 13720 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.004s
I20260928 02:11:29.635687 13720 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790561432251112-7085-0/minicluster-data/master-1/data,/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790561432251112-7085-0/minicluster-data/master-1/wal
uuid: "bafe67917b9a4b40a60409c753bf4b96"
format_stamp: "Formatted at 2026-09-28 02:11:29 on dist-test-slave-86gv"
server_key: "c5221606d2f99f636c600e995e76e14c"
server_key_iv: "d4eaff2fdba8a601b8abafbe6a9d7389"
server_key_version: "encryptionkey@0"
I20260928 02:11:29.635906 13720 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790561432251112-7085-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790561432251112-7085-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790561432251112-7085-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:11:29.663731 13720 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:11:29.668061 13720 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:11:29.668357 13720 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:11:29.687017 13720 rpc_server.cc:307] RPC server started. Bound to: 127.6.235.125:41279
I20260928 02:11:29.687106 13786 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.235.125:41279 every 8 connection(s)
I20260928 02:11:29.688427 13720 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790561432251112-7085-0/minicluster-data/master-1/data/info.pb
I20260928 02:11:29.692252 13787 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:11:29.695147 7085 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTlaUuc/build/asan/bin/kudu as pid 13720
I20260928 02:11:29.700037 13787 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.6.235.126" port: 34853 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 02:11:29 dist-test-slave-86gv krb5kdc[13212](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561486, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.6.235.125@KRBTEST.COM
Sep 28 02:11:29 dist-test-slave-86gv krb5kdc[13212](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561489, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.235.125@KRBTEST.COM for oryx/127.6.235.126@KRBTEST.COM
I20260928 02:11:29.719885 13791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:11:29.703266 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42076 (local address 127.6.235.126:34853)
0928 02:11:29.703793 (+ 527us) server_negotiation.cc:206] Beginning negotiation
0928 02:11:29.703799 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 02:11:29.704579 (+ 780us) server_negotiation.cc:407] Connection header received
0928 02:11:29.704734 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:11:29.704740 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:11:29.704816 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:11:29.704962 (+ 146us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:11:29.706383 (+ 1421us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:11:29.707409 (+ 1026us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:11:29.709442 (+ 2033us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:11:29.709914 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:11:29.710020 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:11:29.713848 (+ 3828us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:11:29.713853 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:11:29.713856 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:11:29.713910 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:11:29.716773 (+ 2863us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:11:29.717662 (+ 889us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:11:29.717666 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:11:29.717669 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:11:29.717829 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:11:29.718263 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:11:29.718268 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:11:29.718271 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:11:29.718699 (+ 428us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:11:29.718846 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0928 02:11:29.719477 (+ 631us) server_negotiation.cc:299] Negotiation successful
0928 02:11:29.719649 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":383,"thread_start_us":211,"threads_started":1}
I20260928 02:11:29.721261 13790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:11:29.697381 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43092 (local address 127.6.235.125:41279)
0928 02:11:29.698150 (+ 769us) server_negotiation.cc:206] Beginning negotiation
0928 02:11:29.698165 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0928 02:11:29.698213 (+ 48us) server_negotiation.cc:407] Connection header received
0928 02:11:29.700453 (+ 2240us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:11:29.700485 (+ 32us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:11:29.701060 (+ 575us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:11:29.701511 (+ 451us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:11:29.702876 (+ 1365us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:11:29.704922 (+ 2046us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:11:29.706690 (+ 1768us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:11:29.707706 (+ 1016us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:11:29.707846 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:11:29.712656 (+ 4810us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:11:29.712665 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:11:29.712680 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:11:29.712711 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:11:29.715806 (+ 3095us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:11:29.716514 (+ 708us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:11:29.716525 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:11:29.716538 (+ 13us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:11:29.716732 (+ 194us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:11:29.717201 (+ 469us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:11:29.717205 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:11:29.717208 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:11:29.717910 (+ 702us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:11:29.718217 (+ 307us) server_negotiation.cc:1035] Waiting for connection context
0928 02:11:29.718755 (+ 538us) server_negotiation.cc:299] Negotiation successful
0928 02:11:29.719438 (+ 683us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":531,"thread_start_us":325,"threads_started":1}
I20260928 02:11:29.721290 13792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:11:29.703429 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.235.126:34853 (local address 127.0.0.1:42076)
0928 02:11:29.704426 (+ 997us) negotiation.cc:107] Waiting for socket to connect
0928 02:11:29.704466 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0928 02:11:29.704610 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:11:29.705069 (+ 459us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:11:29.705075 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:11:29.705917 (+ 842us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:11:29.706228 (+ 311us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:11:29.706240 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:11:29.707553 (+ 1313us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:11:29.707557 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:11:29.709285 (+ 1728us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:11:29.709301 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:11:29.710164 (+ 863us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:11:29.710173 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:11:29.710394 (+ 221us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:11:29.711063 (+ 669us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:11:29.711095 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:11:29.713617 (+ 2522us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:11:29.716935 (+ 3318us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:11:29.716946 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:11:29.716954 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:11:29.717553 (+ 599us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:11:29.717939 (+ 386us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:11:29.717946 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:11:29.717952 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:11:29.718115 (+ 163us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:11:29.718864 (+ 749us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:11:29.718873 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:11:29.719127 (+ 254us) client_negotiation.cc:769] Sending connection context
0928 02:11:29.719383 (+ 256us) client_negotiation.cc:240] Negotiation successful
0928 02:11:29.719730 (+ 347us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":721,"mutex_wait_us":1,"thread_start_us":566,"threads_started":1}
I20260928 02:11:29.723484 13787 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.6.235.125" port: 41279 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 02:11:29 dist-test-slave-86gv krb5kdc[13212](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561489, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.235.125@KRBTEST.COM for oryx/127.6.235.125@KRBTEST.COM
I20260928 02:11:29.735523 13792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:11:29.724349 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.235.125:41279 (local address 127.0.0.1:43104)
0928 02:11:29.724708 (+ 359us) negotiation.cc:107] Waiting for socket to connect
0928 02:11:29.724728 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0928 02:11:29.724830 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:11:29.725171 (+ 341us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:11:29.725174 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:11:29.725521 (+ 347us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:11:29.725634 (+ 113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:11:29.725641 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:11:29.726591 (+ 950us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:11:29.726596 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:11:29.727442 (+ 846us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:11:29.727452 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:11:29.728096 (+ 644us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:11:29.728102 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:11:29.728292 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:11:29.728869 (+ 577us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:11:29.728885 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:11:29.730851 (+ 1966us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:11:29.733519 (+ 2668us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:11:29.733523 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:11:29.733526 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:11:29.733865 (+ 339us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:11:29.734210 (+ 345us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:11:29.734214 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:11:29.734216 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:11:29.734334 (+ 118us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:11:29.734827 (+ 493us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:11:29.734830 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:11:29.734937 (+ 107us) client_negotiation.cc:769] Sending connection context
0928 02:11:29.735040 (+ 103us) client_negotiation.cc:240] Negotiation successful
0928 02:11:29.735274 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":262}
I20260928 02:11:29.735714 13790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:11:29.724548 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43104 (local address 127.6.235.125:41279)
0928 02:11:29.724710 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0928 02:11:29.724717 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 02:11:29.724775 (+ 58us) server_negotiation.cc:407] Connection header received
0928 02:11:29.724994 (+ 219us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:11:29.724999 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:11:29.725062 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:11:29.725155 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:11:29.725768 (+ 613us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:11:29.726456 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:11:29.727575 (+ 1119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:11:29.727963 (+ 388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:11:29.728090 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:11:29.731022 (+ 2932us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:11:29.731027 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:11:29.731030 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:11:29.731058 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:11:29.733400 (+ 2342us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:11:29.733966 (+ 566us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:11:29.733969 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:11:29.733972 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:11:29.734074 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:11:29.734433 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:11:29.734437 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:11:29.734439 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:11:29.734731 (+ 292us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:11:29.734837 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0928 02:11:29.735120 (+ 283us) server_negotiation.cc:299] Negotiation successful
0928 02:11:29.735394 (+ 274us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":104,"server-negotiator.queue_time_us":53}
I20260928 02:11:29.740420 13787 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bafe67917b9a4b40a60409c753bf4b96: Bootstrap starting.
I20260928 02:11:29.743533 13787 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bafe67917b9a4b40a60409c753bf4b96: Neither blocks nor log segments found. Creating new log.
I20260928 02:11:29.744576 13787 log.cc:824] T 00000000000000000000000000000000 P bafe67917b9a4b40a60409c753bf4b96: Log is configured to *not* fsync() on all Append() calls
I20260928 02:11:29.747336 13787 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bafe67917b9a4b40a60409c753bf4b96: No bootstrap required, opened a new log
I20260928 02:11:29.754324 13787 raft_consensus.cc:348] T 00000000000000000000000000000000 P bafe67917b9a4b40a60409c753bf4b96 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "11adc61886584956adfdac31047373c1" member_type: VOTER last_known_addr { host: "127.6.235.126" port: 34853 } } peers { permanent_uuid: "bafe67917b9a4b40a60409c753bf4b96" member_type: VOTER last_known_addr { host: "127.6.235.125" port: 41279 } }
I20260928 02:11:29.754666 13787 raft_consensus.cc:374] T 00000000000000000000000000000000 P bafe67917b9a4b40a60409c753bf4b96 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:11:29.754776 13787 raft_consensus.cc:729] T 00000000000000000000000000000000 P bafe67917b9a4b40a60409c753bf4b96 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bafe67917b9a4b40a60409c753bf4b96, State: Initialized, Role: FOLLOWER
I20260928 02:11:29.755347 13787 consensus_queue.cc:260] T 00000000000000000000000000000000 P bafe67917b9a4b40a60409c753bf4b96 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "11adc61886584956adfdac31047373c1" member_type: VOTER last_known_addr { host: "127.6.235.126" port: 34853 } } peers { permanent_uuid: "bafe67917b9a4b40a60409c753bf4b96" member_type: VOTER last_known_addr { host: "127.6.235.125" port: 41279 } }
I20260928 02:11:29.756158 13794 sys_catalog.cc:455] T 00000000000000000000000000000000 P bafe67917b9a4b40a60409c753bf4b96 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "11adc61886584956adfdac31047373c1" member_type: VOTER last_known_addr { host: "127.6.235.126" port: 34853 } } peers { permanent_uuid: "bafe67917b9a4b40a60409c753bf4b96" member_type: VOTER last_known_addr { host: "127.6.235.125" port: 41279 } } }
I20260928 02:11:29.756464 13794 sys_catalog.cc:458] T 00000000000000000000000000000000 P bafe67917b9a4b40a60409c753bf4b96 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 02:11:29.757248 13787 sys_catalog.cc:564] T 00000000000000000000000000000000 P bafe67917b9a4b40a60409c753bf4b96 [sys.catalog]: configured and running, proceeding with master startup.
W20260928 02:11:29.767243 13805 catalog_manager.cc:1558] T 00000000000000000000000000000000 P bafe67917b9a4b40a60409c753bf4b96: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 02:11:29.767437 13805 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 02:11:29.769790 13720 master_runner.cc:428] Detected that this master bafe67917b9a4b40a60409c753bf4b96 is joining an existing cluster
I20260928 02:11:29.769892 13720 master_runner.cc:432] Initiating AddMaster RPC to add 127.6.235.125:41279
I20260928 02:11:29.796157 13791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:11:29.774132 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42084 (local address 127.6.235.126:34853)
0928 02:11:29.774315 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0928 02:11:29.774324 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 02:11:29.779951 (+ 5627us) server_negotiation.cc:407] Connection header received
0928 02:11:29.780085 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:11:29.780091 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:11:29.780193 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:11:29.780299 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:11:29.781840 (+ 1541us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:11:29.782898 (+ 1058us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:11:29.784990 (+ 2092us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:11:29.785638 (+ 648us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:11:29.785757 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:11:29.788459 (+ 2702us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:11:29.788466 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:11:29.788471 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:11:29.788510 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:11:29.792042 (+ 3532us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:11:29.793466 (+ 1424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:11:29.793471 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:11:29.793475 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:11:29.793632 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:11:29.794091 (+ 459us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:11:29.794097 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:11:29.794103 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:11:29.794781 (+ 678us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:11:29.794931 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0928 02:11:29.795544 (+ 613us) server_negotiation.cc:299] Negotiation successful
0928 02:11:29.795798 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260928 02:11:29.799489 13811 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:11:29.774258 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.235.126:34853 (local address 127.0.0.1:42084)
0928 02:11:29.779804 (+ 5546us) negotiation.cc:107] Waiting for socket to connect
0928 02:11:29.779830 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0928 02:11:29.779975 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:11:29.780479 (+ 504us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:11:29.780485 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:11:29.781218 (+ 733us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:11:29.781648 (+ 430us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:11:29.781662 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:11:29.783110 (+ 1448us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:11:29.783115 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:11:29.784800 (+ 1685us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:11:29.784815 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:11:29.786074 (+ 1259us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:11:29.786081 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:11:29.786266 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:11:29.786982 (+ 716us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:11:29.787005 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:11:29.788223 (+ 1218us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:11:29.792764 (+ 4541us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:11:29.792774 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:11:29.792779 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:11:29.793319 (+ 540us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:11:29.793788 (+ 469us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:11:29.793796 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:11:29.793801 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:11:29.793964 (+ 163us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:11:29.795017 (+ 1053us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:11:29.795022 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:11:29.795249 (+ 227us) client_negotiation.cc:769] Sending connection context
0928 02:11:29.796667 (+ 1418us) client_negotiation.cc:240] Negotiation successful
0928 02:11:29.798983 (+ 2316us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5383,"thread_start_us":5238,"threads_started":1}
I20260928 02:11:29.801829 13790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:11:29.774227 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43110 (local address 127.6.235.125:41279)
0928 02:11:29.774429 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0928 02:11:29.774437 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 02:11:29.787466 (+ 13029us) server_negotiation.cc:407] Connection header received
0928 02:11:29.787886 (+ 420us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:11:29.787897 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:11:29.787971 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:11:29.788046 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:11:29.789804 (+ 1758us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:11:29.791230 (+ 1426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:11:29.792700 (+ 1470us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:11:29.793477 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:11:29.793587 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:11:29.795929 (+ 2342us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:11:29.795935 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:11:29.795940 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:11:29.795976 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:11:29.798792 (+ 2816us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:11:29.799623 (+ 831us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:11:29.799634 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:11:29.799639 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:11:29.799772 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:11:29.800213 (+ 441us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:11:29.800217 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:11:29.800220 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:11:29.800516 (+ 296us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:11:29.800617 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0928 02:11:29.801017 (+ 400us) server_negotiation.cc:299] Negotiation successful
0928 02:11:29.801372 (+ 355us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":148,"server-negotiator.queue_time_us":63}
I20260928 02:11:29.802605 13812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:11:29.780305 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.235.125:41279 (local address 127.0.0.1:43110)
0928 02:11:29.787364 (+ 7059us) negotiation.cc:107] Waiting for socket to connect
0928 02:11:29.787390 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0928 02:11:29.787764 (+ 374us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:11:29.788247 (+ 483us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:11:29.788253 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:11:29.788993 (+ 740us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:11:29.789641 (+ 648us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:11:29.789656 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:11:29.791464 (+ 1808us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:11:29.791468 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:11:29.792553 (+ 1085us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:11:29.792565 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:11:29.793855 (+ 1290us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:11:29.793861 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:11:29.794033 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:11:29.794665 (+ 632us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:11:29.794689 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:11:29.795752 (+ 1063us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:11:29.798980 (+ 3228us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:11:29.798985 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:11:29.798988 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:11:29.799497 (+ 509us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:11:29.799900 (+ 403us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:11:29.799904 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:11:29.799907 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:11:29.800090 (+ 183us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:11:29.800678 (+ 588us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:11:29.800682 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:11:29.800774 (+ 92us) client_negotiation.cc:769] Sending connection context
0928 02:11:29.800925 (+ 151us) client_negotiation.cc:240] Negotiation successful
0928 02:11:29.802071 (+ 1146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6889,"thread_start_us":7160,"threads_started":1}
WARNING: no policy specified for oryx/127.6.235.124@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.6.235.124@KRBTEST.COM" created.
Sep 28 02:11:29 dist-test-slave-86gv krb5kdc[13212](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561487, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.235.126@KRBTEST.COM for oryx/127.6.235.125@KRBTEST.COM
I20260928 02:11:29.824935 13790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:11:29.806980 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43114 (local address 127.6.235.125:41279)
0928 02:11:29.807165 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0928 02:11:29.807173 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 02:11:29.807848 (+ 675us) server_negotiation.cc:407] Connection header received
0928 02:11:29.808234 (+ 386us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:11:29.808239 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:11:29.808358 (+ 119us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:11:29.808474 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:11:29.809849 (+ 1375us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:11:29.810664 (+ 815us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:11:29.812641 (+ 1977us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:11:29.813389 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:11:29.813499 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:11:29.818150 (+ 4651us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:11:29.818158 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:11:29.818163 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:11:29.818204 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:11:29.822024 (+ 3820us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:11:29.822803 (+ 779us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:11:29.822808 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:11:29.822822 (+ 14us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:11:29.822972 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:11:29.823376 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:11:29.823381 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:11:29.823386 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:11:29.823661 (+ 275us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:11:29.823822 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0928 02:11:29.824294 (+ 472us) server_negotiation.cc:299] Negotiation successful
0928 02:11:29.824521 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260928 02:11:29.824981 13814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:11:29.806986 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.235.125:41279 (local address 127.0.0.1:43114)
0928 02:11:29.807735 (+ 749us) negotiation.cc:107] Waiting for socket to connect
0928 02:11:29.807773 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0928 02:11:29.807973 (+ 200us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:11:29.808517 (+ 544us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:11:29.808525 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:11:29.809410 (+ 885us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:11:29.809687 (+ 277us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:11:29.809698 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:11:29.810811 (+ 1113us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:11:29.810818 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:11:29.812471 (+ 1653us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:11:29.812486 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:11:29.813805 (+ 1319us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:11:29.813811 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:11:29.814032 (+ 221us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:11:29.814775 (+ 743us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:11:29.814801 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:11:29.817919 (+ 3118us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:11:29.822202 (+ 4283us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:11:29.822213 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:11:29.822219 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:11:29.822648 (+ 429us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:11:29.823093 (+ 445us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:11:29.823100 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:11:29.823103 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:11:29.823252 (+ 149us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:11:29.823812 (+ 560us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:11:29.823821 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:11:29.824001 (+ 180us) client_negotiation.cc:769] Sending connection context
0928 02:11:29.824191 (+ 190us) client_negotiation.cc:240] Negotiation successful
0928 02:11:29.824521 (+ 330us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":522,"thread_start_us":366,"threads_started":1}
I20260928 02:11:29.827059 13257 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.6.235.125:41279: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "bafe67917b9a4b40a60409c753bf4b96"
member_type: NON_VOTER
last_known_addr {
host: "127.6.235.125"
port: 41279
}
attrs {
promote: true
}
}
dest_uuid: "11adc61886584956adfdac31047373c1"
cas_config_opid_index: -1
I20260928 02:11:29.828051 13257 consensus_queue.cc:237] T 00000000000000000000000000000000 P 11adc61886584956adfdac31047373c1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "11adc61886584956adfdac31047373c1" member_type: VOTER last_known_addr { host: "127.6.235.126" port: 34853 } } peers { permanent_uuid: "bafe67917b9a4b40a60409c753bf4b96" member_type: NON_VOTER last_known_addr { host: "127.6.235.125" port: 41279 } attrs { promote: true } }
I20260928 02:11:29.836293 13762 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bafe67917b9a4b40a60409c753bf4b96 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:11:29.835999 13818 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 11adc61886584956adfdac31047373c1 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER bafe67917b9a4b40a60409c753bf4b96 (127.6.235.125) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "11adc61886584956adfdac31047373c1" member_type: VOTER last_known_addr { host: "127.6.235.126" port: 34853 } } peers { permanent_uuid: "bafe67917b9a4b40a60409c753bf4b96" member_type: NON_VOTER last_known_addr { host: "127.6.235.125" port: 41279 } attrs { promote: true } } }
I20260928 02:11:29.840219 13818 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.6.235.125:41279
I20260928 02:11:29.840674 13762 raft_consensus.cc:1261] T 00000000000000000000000000000000 P bafe67917b9a4b40a60409c753bf4b96 [term 1 FOLLOWER]: Refusing update from remote peer 11adc61886584956adfdac31047373c1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20260928 02:11:29.841061 13817 sys_catalog.cc:455] T 00000000000000000000000000000000 P 11adc61886584956adfdac31047373c1 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "11adc61886584956adfdac31047373c1" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "11adc61886584956adfdac31047373c1" member_type: VOTER last_known_addr { host: "127.6.235.126" port: 34853 } } peers { permanent_uuid: "bafe67917b9a4b40a60409c753bf4b96" member_type: NON_VOTER last_known_addr { host: "127.6.235.125" port: 41279 } attrs { promote: true } } }
I20260928 02:11:29.841274 13720 master.cc:537] Master@127.6.235.125:41279 shutting down...
I20260928 02:11:29.841358 13817 sys_catalog.cc:458] T 00000000000000000000000000000000 P 11adc61886584956adfdac31047373c1 [sys.catalog]: This master's current role is: LEADER
I20260928 02:11:29.843811 13817 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 11adc61886584956adfdac31047373c1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bafe67917b9a4b40a60409c753bf4b96" member_type: NON_VOTER last_known_addr { host: "127.6.235.125" port: 41279 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 0, Time since last communication: 0.000s
W20260928 02:11:29.847584 13244 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master
W20260928 02:11:29.849515 13244 consensus_peers.cc:487] T 00000000000000000000000000000000 P 11adc61886584956adfdac31047373c1 -> Peer bafe67917b9a4b40a60409c753bf4b96 (127.6.235.125:41279): Couldn't send request to peer bafe67917b9a4b40a60409c753bf4b96. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.6.235.124 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790561432251112-7085-0/krb5kdc/oryx_127.6.235.124.keytab.
Entry for principal oryx/127.6.235.124 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790561432251112-7085-0/krb5kdc/oryx_127.6.235.124.keytab.
I20260928 02:11:29.851847 7085 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTlaUuc/build/asan/bin/kudu
/tmp/dist-test-taskTlaUuc/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790561432251112-7085-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790561432251112-7085-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790561432251112-7085-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790561432251112-7085-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.235.124:33005
--webserver_interface=127.6.235.124
--webserver_port=0
--keytab_file=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790561432251112-7085-0/krb5kdc/oryx_127.6.235.124.keytab
--principal=oryx/127.6.235.124
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.235.84:44519
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.235.126:34853,127.6.235.125:41279,127.6.235.124:33005
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790561432251112-7085-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790561432251112-7085-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790561432251112-7085-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 02:11:29.853066 13720 raft_consensus.cc:2229] T 00000000000000000000000000000000 P bafe67917b9a4b40a60409c753bf4b96 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 02:11:29.853511 13720 raft_consensus.cc:2258] T 00000000000000000000000000000000 P bafe67917b9a4b40a60409c753bf4b96 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 02:11:29.853696 13720 tablet_replica.cc:330] T 00000000000000000000000000000000 P bafe67917b9a4b40a60409c753bf4b96: stopping tablet replica
I20260928 02:11:29.861595 13720 master.cc:552] Master@127.6.235.125:41279 shutdown complete.
I20260928 02:11:29.861716 13720 master_runner.cc:305] Clearing existing system tablet
I20260928 02:11:29.862084 13720 fs_manager.cc:359] Metadata directory not provided
I20260928 02:11:29.862201 13720 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:11:29.864557 13720 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.001s
I20260928 02:11:29.866106 13823 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:11:29.866433 13720 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790561432251112-7085-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:11:29.866711 13720 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260928 02:11:29.866789 13720 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790561432251112-7085-0/minicluster-data/master-1/data,/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790561432251112-7085-0/minicluster-data/master-1/wal
uuid: "bafe67917b9a4b40a60409c753bf4b96"
format_stamp: "Formatted at 2026-09-28 02:11:29 on dist-test-slave-86gv"
server_key: "c5221606d2f99f636c600e995e76e14c"
server_key_iv: "d4eaff2fdba8a601b8abafbe6a9d7389"
server_key_version: "encryptionkey@0"
I20260928 02:11:29.870959 13720 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P bafe67917b9a4b40a60409c753bf4b96: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 02:11:29.875922 13720 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P bafe67917b9a4b40a60409c753bf4b96: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260928 02:11:29.876123 13720 log.cc:1196] T 00000000000000000000000000000000 P bafe67917b9a4b40a60409c753bf4b96: Deleting WAL directory at /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790561432251112-7085-0/minicluster-data/master-1/wal/wals/00000000000000000000000000000000
I20260928 02:11:29.876544 13720 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P bafe67917b9a4b40a60409c753bf4b96: Deleting consensus metadata
I20260928 02:11:29.876804 13720 master_runner.cc:315] Copying system tablet from 127.6.235.126:34853
I20260928 02:11:29.879316 13720 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P bafe67917b9a4b40a60409c753bf4b96: tablet copy: Beginning tablet copy session from remote peer at address 127.6.235.126:34853
I20260928 02:11:29.892091 13791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:11:29.880108 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42096 (local address 127.6.235.126:34853)
0928 02:11:29.880324 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0928 02:11:29.880334 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 02:11:29.880856 (+ 522us) server_negotiation.cc:407] Connection header received
0928 02:11:29.881175 (+ 319us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:11:29.881182 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:11:29.881267 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:11:29.881326 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:11:29.882265 (+ 939us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:11:29.882981 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:11:29.884296 (+ 1315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:11:29.884727 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:11:29.884865 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:11:29.886738 (+ 1873us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:11:29.886744 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:11:29.886749 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:11:29.886792 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:11:29.889491 (+ 2699us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:11:29.890170 (+ 679us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:11:29.890185 (+ 15us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:11:29.890194 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:11:29.890321 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:11:29.890688 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:11:29.890695 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:11:29.890700 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:11:29.890969 (+ 269us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:11:29.891124 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0928 02:11:29.891496 (+ 372us) server_negotiation.cc:299] Negotiation successful
0928 02:11:29.891712 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260928 02:11:29.892141 13829 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:11:29.880061 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.235.126:34853 (local address 127.0.0.1:42096)
0928 02:11:29.880713 (+ 652us) negotiation.cc:107] Waiting for socket to connect
0928 02:11:29.880732 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0928 02:11:29.881006 (+ 274us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:11:29.881490 (+ 484us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:11:29.881493 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:11:29.881833 (+ 340us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:11:29.882108 (+ 275us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:11:29.882122 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:11:29.883129 (+ 1007us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:11:29.883135 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:11:29.884143 (+ 1008us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:11:29.884155 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:11:29.884877 (+ 722us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:11:29.884882 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:11:29.885132 (+ 250us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:11:29.885818 (+ 686us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:11:29.885836 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:11:29.886525 (+ 689us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:11:29.889631 (+ 3106us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:11:29.889637 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:11:29.889642 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:11:29.890050 (+ 408us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:11:29.890436 (+ 386us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:11:29.890441 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:11:29.890445 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:11:29.890582 (+ 137us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:11:29.891100 (+ 518us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:11:29.891105 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:11:29.891268 (+ 163us) client_negotiation.cc:769] Sending connection context
0928 02:11:29.891407 (+ 139us) client_negotiation.cc:240] Negotiation successful
0928 02:11:29.891722 (+ 315us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":526,"thread_start_us":406,"threads_started":1}
I20260928 02:11:29.893368 13277 tablet_copy_service.cc:140] P 11adc61886584956adfdac31047373c1: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer bafe67917b9a4b40a60409c753bf4b96 ({username='oryx', principal='oryx/127.6.235.125@KRBTEST.COM'} at 127.0.0.1:42096)
I20260928 02:11:29.893647 13277 tablet_copy_service.cc:161] P 11adc61886584956adfdac31047373c1: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer bafe67917b9a4b40a60409c753bf4b96 at {username='oryx', principal='oryx/127.6.235.125@KRBTEST.COM'} at 127.0.0.1:42096: session id = bafe67917b9a4b40a60409c753bf4b96-00000000000000000000000000000000
I20260928 02:11:29.896682 13277 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 11adc61886584956adfdac31047373c1: Tablet Copy: opened 0 blocks and 1 log segments
I20260928 02:11:29.898684 13720 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260928 02:11:29.903429 13720 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P bafe67917b9a4b40a60409c753bf4b96: tablet copy: Starting download of 0 data blocks...
I20260928 02:11:29.903744 13720 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P bafe67917b9a4b40a60409c753bf4b96: tablet copy: Starting download of 1 WAL segments...
I20260928 02:11:29.906237 13720 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P bafe67917b9a4b40a60409c753bf4b96: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260928 02:11:29.909715 13277 tablet_copy_service.cc:342] P 11adc61886584956adfdac31047373c1: Request end of tablet copy session bafe67917b9a4b40a60409c753bf4b96-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.6.235.125@KRBTEST.COM'} at 127.0.0.1:42096
I20260928 02:11:29.909930 13277 tablet_copy_service.cc:433] P 11adc61886584956adfdac31047373c1: ending tablet copy session bafe67917b9a4b40a60409c753bf4b96-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer bafe67917b9a4b40a60409c753bf4b96
I20260928 02:11:29.920738 13720 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:11:29.923744 13830 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:11:29.923960 13831 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:11:29.925734 13833 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:11:29.926086 13720 server_base.cc:1034] running on GCE node
I20260928 02:11:29.926394 13720 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:11:29.927007 13720 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:11:29.928226 13720 hybrid_clock.cc:648] HybridClock initialized: now 1790561489928155 us; error 38 us; skew 500 ppm
I20260928 02:11:29.928447 13720 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 02:11:29 dist-test-slave-86gv krb5kdc[13212](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561489, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.235.125@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:11:29.930637 13720 init.cc:377] Logged in from keytab as oryx/127.6.235.125@KRBTEST.COM (short username oryx)
I20260928 02:11:29.931880 13720 webserver.cc:460] Webserver started at http://127.6.235.125:40267/ using document root <none> and password file <none>
I20260928 02:11:29.932221 13720 fs_manager.cc:359] Metadata directory not provided
I20260928 02:11:29.932308 13720 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:11:29.935513 13720 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s
I20260928 02:11:29.937493 13840 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:11:29.937947 13720 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.002s sys 0.000s
I20260928 02:11:29.938133 13720 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790561432251112-7085-0/minicluster-data/master-1/data,/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790561432251112-7085-0/minicluster-data/master-1/wal
uuid: "bafe67917b9a4b40a60409c753bf4b96"
format_stamp: "Formatted at 2026-09-28 02:11:29 on dist-test-slave-86gv"
server_key: "c5221606d2f99f636c600e995e76e14c"
server_key_iv: "d4eaff2fdba8a601b8abafbe6a9d7389"
server_key_version: "encryptionkey@0"
I20260928 02:11:29.938383 13720 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790561432251112-7085-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790561432251112-7085-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790561432251112-7085-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:11:29.966248 13720 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:11:29.971166 13720 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:11:30.006104 13720 rpc_server.cc:307] RPC server started. Bound to: 127.6.235.125:41279
I20260928 02:11:30.006155 13891 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.235.125:41279 every 8 connection(s)
I20260928 02:11:30.007295 13720 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790561432251112-7085-0/minicluster-data/master-1/data/info.pb
I20260928 02:11:30.013222 13892 sys_catalog.cc:263] Verifying existing consensus state
I20260928 02:11:30.016990 13892 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bafe67917b9a4b40a60409c753bf4b96: Bootstrap starting.
I20260928 02:11:30.038825 13892 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bafe67917b9a4b40a60409c753bf4b96: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 02:11:30.039470 13892 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bafe67917b9a4b40a60409c753bf4b96: Bootstrap complete.
I20260928 02:11:30.040820 13892 raft_consensus.cc:348] T 00000000000000000000000000000000 P bafe67917b9a4b40a60409c753bf4b96 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "11adc61886584956adfdac31047373c1" member_type: VOTER last_known_addr { host: "127.6.235.126" port: 34853 } } peers { permanent_uuid: "bafe67917b9a4b40a60409c753bf4b96" member_type: NON_VOTER last_known_addr { host: "127.6.235.125" port: 41279 } attrs { promote: true } }
I20260928 02:11:30.041162 13892 raft_consensus.cc:729] T 00000000000000000000000000000000 P bafe67917b9a4b40a60409c753bf4b96 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: bafe67917b9a4b40a60409c753bf4b96, State: Initialized, Role: LEARNER
I20260928 02:11:30.041414 13892 consensus_queue.cc:260] T 00000000000000000000000000000000 P bafe67917b9a4b40a60409c753bf4b96 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "11adc61886584956adfdac31047373c1" member_type: VOTER last_known_addr { host: "127.6.235.126" port: 34853 } } peers { permanent_uuid: "bafe67917b9a4b40a60409c753bf4b96" member_type: NON_VOTER last_known_addr { host: "127.6.235.125" port: 41279 } attrs { promote: true } }
I20260928 02:11:30.042241 13895 sys_catalog.cc:455] T 00000000000000000000000000000000 P bafe67917b9a4b40a60409c753bf4b96 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "11adc61886584956adfdac31047373c1" member_type: VOTER last_known_addr { host: "127.6.235.126" port: 34853 } } peers { permanent_uuid: "bafe67917b9a4b40a60409c753bf4b96" member_type: NON_VOTER last_known_addr { host: "127.6.235.125" port: 41279 } attrs { promote: true } } }
I20260928 02:11:30.042498 13892 sys_catalog.cc:564] T 00000000000000000000000000000000 P bafe67917b9a4b40a60409c753bf4b96 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 02:11:30.042634 13895 sys_catalog.cc:458] T 00000000000000000000000000000000 P bafe67917b9a4b40a60409c753bf4b96 [sys.catalog]: This master's current role is: LEARNER
I20260928 02:11:30.050083 13906 catalog_manager.cc:1259] Loaded cluster ID: 33ac180e627642b2a4850f9fab0c8be0
I20260928 02:11:30.050190 13906 catalog_manager.cc:1552] T 00000000000000000000000000000000 P bafe67917b9a4b40a60409c753bf4b96: loading cluster ID for follower catalog manager: success
I20260928 02:11:30.053781 13906 catalog_manager.cc:1574] T 00000000000000000000000000000000 P bafe67917b9a4b40a60409c753bf4b96: acquiring CA information for follower catalog manager: success
I20260928 02:11:30.056347 13906 catalog_manager.cc:1602] T 00000000000000000000000000000000 P bafe67917b9a4b40a60409c753bf4b96: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260928 02:11:30.160497 13714 heartbeater.cc:503] Master 127.6.235.126:34853 was elected leader, sending a full tablet report...
W20260928 02:11:30.243894 13821 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:11:30.244196 13821 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:11:30.244370 13821 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:11:30.251912 13821 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 02:11:30.252035 13821 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:11:30.252118 13821 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:11:30.252221 13821 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 02:11:30.252281 13821 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 02:11:30.262791 13821 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.235.84:44519
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790561432251112-7085-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790561432251112-7085-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.6.235.126:34853,127.6.235.125:41279,127.6.235.124:33005
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790561432251112-7085-0/krb5kdc/oryx_127.6.235.124.keytab
--principal=oryx/127.6.235.124
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.235.124:33005
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790561432251112-7085-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.6.235.124
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790561432251112-7085-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 00:51:59 UTC on ccycloud.quasar-axgkwn.root.comops.site
build id 4676464
ASAN enabled
I20260928 02:11:30.264277 13821 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:11:30.265789 13821 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:11:30.278927 13911 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:11:30.278997 13912 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:11:30.279036 13914 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:11:30.280409 13821 server_base.cc:1034] running on GCE node
I20260928 02:11:30.281445 13821 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:11:30.283218 13821 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:11:30.284462 13821 hybrid_clock.cc:648] HybridClock initialized: now 1790561490284374 us; error 93 us; skew 500 ppm
I20260928 02:11:30.284850 13821 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 02:11:30 dist-test-slave-86gv krb5kdc[13212](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561490, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.235.124@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:11:30.288302 13821 init.cc:377] Logged in from keytab as oryx/127.6.235.124@KRBTEST.COM (short username oryx)
I20260928 02:11:30.289889 13821 webserver.cc:460] Webserver started at http://127.6.235.124:45327/ using document root <none> and password file <none>
I20260928 02:11:30.290684 13821 fs_manager.cc:359] Metadata directory not provided
I20260928 02:11:30.290796 13821 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:11:30.291064 13821 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:11:30.294329 13821 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790561432251112-7085-0/minicluster-data/master-2/data/instance:
uuid: "254f36e26f8d4ddf823cf81c7a5c2986"
format_stamp: "Formatted at 2026-09-28 02:11:30 on dist-test-slave-86gv"
server_key: "64ad8e8c867224fbcc7569514b56e3a1"
server_key_iv: "120dfcb6e992c2672c0f57db7f35d68d"
server_key_version: "encryptionkey@0"
I20260928 02:11:30.295080 13821 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790561432251112-7085-0/minicluster-data/master-2/wal/instance:
uuid: "254f36e26f8d4ddf823cf81c7a5c2986"
format_stamp: "Formatted at 2026-09-28 02:11:30 on dist-test-slave-86gv"
server_key: "64ad8e8c867224fbcc7569514b56e3a1"
server_key_iv: "120dfcb6e992c2672c0f57db7f35d68d"
server_key_version: "encryptionkey@0"
I20260928 02:11:30.300474 13821 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260928 02:11:30.304260 13921 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:11:30.305900 13821 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260928 02:11:30.306077 13821 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790561432251112-7085-0/minicluster-data/master-2/data,/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790561432251112-7085-0/minicluster-data/master-2/wal
uuid: "254f36e26f8d4ddf823cf81c7a5c2986"
format_stamp: "Formatted at 2026-09-28 02:11:30 on dist-test-slave-86gv"
server_key: "64ad8e8c867224fbcc7569514b56e3a1"
server_key_iv: "120dfcb6e992c2672c0f57db7f35d68d"
server_key_version: "encryptionkey@0"
I20260928 02:11:30.306293 13821 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790561432251112-7085-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790561432251112-7085-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790561432251112-7085-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:11:30.331405 13821 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:11:30.336120 13821 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:11:30.336421 13821 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:11:30.356247 13821 rpc_server.cc:307] RPC server started. Bound to: 127.6.235.124:33005
I20260928 02:11:30.356281 13972 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.235.124:33005 every 8 connection(s)
I20260928 02:11:30.357834 13821 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790561432251112-7085-0/minicluster-data/master-2/data/info.pb
I20260928 02:11:30.361498 7085 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTlaUuc/build/asan/bin/kudu as pid 13821
I20260928 02:11:30.362733 13973 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:11:30.373457 13973 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.6.235.126" port: 34853 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 02:11:30 dist-test-slave-86gv krb5kdc[13212](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561486, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.6.235.124@KRBTEST.COM
I20260928 02:11:30.393153 13976 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:11:30.363877 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41442 (local address 127.6.235.124:33005)
0928 02:11:30.364714 (+ 837us) server_negotiation.cc:206] Beginning negotiation
0928 02:11:30.364730 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0928 02:11:30.364783 (+ 53us) server_negotiation.cc:407] Connection header received
0928 02:11:30.366730 (+ 1947us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:11:30.366762 (+ 32us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:11:30.367408 (+ 646us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:11:30.367922 (+ 514us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:11:30.369845 (+ 1923us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:11:30.372042 (+ 2197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:11:30.374048 (+ 2006us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:11:30.375128 (+ 1080us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:11:30.376290 (+ 1162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:11:30.381881 (+ 5591us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:11:30.381898 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:11:30.381929 (+ 31us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:11:30.382010 (+ 81us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:11:30.387874 (+ 5864us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:11:30.388821 (+ 947us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:11:30.388831 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:11:30.388842 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:11:30.389068 (+ 226us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:11:30.389534 (+ 466us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:11:30.389540 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:11:30.389543 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:11:30.390437 (+ 894us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:11:30.390831 (+ 394us) server_negotiation.cc:1035] Waiting for connection context
0928 02:11:30.391267 (+ 436us) server_negotiation.cc:299] Negotiation successful
0928 02:11:30.391691 (+ 424us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":34,"server-negotiator.queue_time_us":587,"thread_start_us":356,"threads_started":1}
Sep 28 02:11:30 dist-test-slave-86gv krb5kdc[13212](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561490, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.235.124@KRBTEST.COM for oryx/127.6.235.126@KRBTEST.COM
I20260928 02:11:30.398713 13979 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:11:30.378590 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.235.125:41279 (local address 127.0.0.1:43126)
0928 02:11:30.391278 (+ 12688us) negotiation.cc:107] Waiting for socket to connect
0928 02:11:30.391299 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0928 02:11:30.391422 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:11:30.392457 (+ 1035us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:11:30.392462 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:11:30.392495 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:11:30.392716 (+ 221us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:11:30.392744 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:11:30.394589 (+ 1845us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:11:30.394596 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:11:30.396603 (+ 2007us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:11:30.396620 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:11:30.397849 (+ 1229us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:11:30.397855 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:11:30.398072 (+ 217us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:11:30.398133 (+ 61us) client_negotiation.cc:769] Sending connection context
0928 02:11:30.398252 (+ 119us) client_negotiation.cc:240] Negotiation successful
0928 02:11:30.398379 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12569,"thread_start_us":12436,"threads_started":1}
I20260928 02:11:30.398738 13980 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:11:30.378706 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43126 (local address 127.6.235.125:41279)
0928 02:11:30.386076 (+ 7370us) server_negotiation.cc:206] Beginning negotiation
0928 02:11:30.386085 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 02:11:30.391411 (+ 5326us) server_negotiation.cc:407] Connection header received
0928 02:11:30.392124 (+ 713us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:11:30.392130 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:11:30.392235 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:11:30.392358 (+ 123us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:11:30.392884 (+ 526us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:11:30.394413 (+ 1529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:11:30.396773 (+ 2360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:11:30.397650 (+ 877us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:11:30.397766 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:11:30.397911 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0928 02:11:30.398299 (+ 388us) server_negotiation.cc:299] Negotiation successful
0928 02:11:30.398452 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7223,"thread_start_us":7108,"threads_started":1}
I20260928 02:11:30.400040 13867 raft_consensus.cc:1203] T 00000000000000000000000000000000 P bafe67917b9a4b40a60409c753bf4b96 [term 1 LEARNER]: Deduplicated request from leader. Original: 0.0->[1.1-1.5] Dedup: 1.5->[]
I20260928 02:11:30.400710 13895 sys_catalog.cc:455] T 00000000000000000000000000000000 P bafe67917b9a4b40a60409c753bf4b96 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 11adc61886584956adfdac31047373c1. Latest consensus state: current_term: 1 leader_uuid: "11adc61886584956adfdac31047373c1" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "11adc61886584956adfdac31047373c1" member_type: VOTER last_known_addr { host: "127.6.235.126" port: 34853 } } peers { permanent_uuid: "bafe67917b9a4b40a60409c753bf4b96" member_type: NON_VOTER last_known_addr { host: "127.6.235.125" port: 41279 } attrs { promote: true } } }
I20260928 02:11:30.401034 13895 sys_catalog.cc:458] T 00000000000000000000000000000000 P bafe67917b9a4b40a60409c753bf4b96 [sys.catalog]: This master's current role is: LEARNER
I20260928 02:11:30.403764 13791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:11:30.377008 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42108 (local address 127.6.235.126:34853)
0928 02:11:30.377219 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0928 02:11:30.377228 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 02:11:30.385469 (+ 8241us) server_negotiation.cc:407] Connection header received
0928 02:11:30.385714 (+ 245us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:11:30.385723 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:11:30.385828 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:11:30.385943 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:11:30.387595 (+ 1652us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:11:30.389104 (+ 1509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:11:30.392353 (+ 3249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:11:30.393038 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:11:30.393194 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:11:30.397503 (+ 4309us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:11:30.397508 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:11:30.397511 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:11:30.397541 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:11:30.401049 (+ 3508us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:11:30.401813 (+ 764us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:11:30.401820 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:11:30.401823 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:11:30.401949 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:11:30.402314 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:11:30.402321 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:11:30.402324 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:11:30.402594 (+ 270us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:11:30.402743 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0928 02:11:30.403231 (+ 488us) server_negotiation.cc:299] Negotiation successful
0928 02:11:30.403439 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260928 02:11:30.403872 13978 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:11:30.376496 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.235.126:34853 (local address 127.0.0.1:42108)
0928 02:11:30.385284 (+ 8788us) negotiation.cc:107] Waiting for socket to connect
0928 02:11:30.385356 (+ 72us) client_negotiation.cc:174] Beginning negotiation
0928 02:11:30.385578 (+ 222us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:11:30.386065 (+ 487us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:11:30.386074 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:11:30.386992 (+ 918us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:11:30.387401 (+ 409us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:11:30.387418 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:11:30.389415 (+ 1997us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:11:30.389424 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:11:30.392197 (+ 2773us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:11:30.392216 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:11:30.393575 (+ 1359us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:11:30.393585 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:11:30.393811 (+ 226us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:11:30.394611 (+ 800us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:11:30.394634 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:11:30.397320 (+ 2686us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:11:30.401199 (+ 3879us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:11:30.401210 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:11:30.401217 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:11:30.401689 (+ 472us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:11:30.402068 (+ 379us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:11:30.402073 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:11:30.402077 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:11:30.402209 (+ 132us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:11:30.402732 (+ 523us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:11:30.402740 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:11:30.402907 (+ 167us) client_negotiation.cc:769] Sending connection context
0928 02:11:30.403145 (+ 238us) client_negotiation.cc:240] Negotiation successful
0928 02:11:30.403462 (+ 317us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8410,"thread_start_us":8185,"threads_started":1}
I20260928 02:11:30.405153 13973 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.6.235.125" port: 41279 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 02:11:30 dist-test-slave-86gv krb5kdc[13212](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561490, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.235.124@KRBTEST.COM for oryx/127.6.235.125@KRBTEST.COM
I20260928 02:11:30.418015 13978 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:11:30.405876 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.235.125:41279 (local address 127.0.0.1:43130)
0928 02:11:30.406049 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0928 02:11:30.406067 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0928 02:11:30.406157 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:11:30.406464 (+ 307us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:11:30.406467 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:11:30.406787 (+ 320us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:11:30.406914 (+ 127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:11:30.406921 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:11:30.407829 (+ 908us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:11:30.407833 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:11:30.408858 (+ 1025us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:11:30.408869 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:11:30.409663 (+ 794us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:11:30.409667 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:11:30.409779 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:11:30.410362 (+ 583us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:11:30.410378 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:11:30.412518 (+ 2140us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:11:30.415619 (+ 3101us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:11:30.415623 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:11:30.415625 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:11:30.416088 (+ 463us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:11:30.416466 (+ 378us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:11:30.416469 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:11:30.416472 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:11:30.416595 (+ 123us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:11:30.417295 (+ 700us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:11:30.417298 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:11:30.417412 (+ 114us) client_negotiation.cc:769] Sending connection context
0928 02:11:30.417552 (+ 140us) client_negotiation.cc:240] Negotiation successful
0928 02:11:30.417798 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":50}
I20260928 02:11:30.418232 13980 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:11:30.405984 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43130 (local address 127.6.235.125:41279)
0928 02:11:30.406139 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0928 02:11:30.406143 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:11:30.406157 (+ 14us) server_negotiation.cc:407] Connection header received
0928 02:11:30.406247 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:11:30.406254 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:11:30.406337 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:11:30.406446 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:11:30.407029 (+ 583us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:11:30.407719 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:11:30.409036 (+ 1317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:11:30.409469 (+ 433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:11:30.409561 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:11:30.412682 (+ 3121us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:11:30.412687 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:11:30.412690 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:11:30.412719 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:11:30.415492 (+ 2773us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:11:30.416212 (+ 720us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:11:30.416216 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:11:30.416219 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:11:30.416350 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:11:30.416727 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:11:30.416732 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:11:30.416734 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:11:30.417176 (+ 442us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:11:30.417301 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0928 02:11:30.417635 (+ 334us) server_negotiation.cc:299] Negotiation successful
0928 02:11:30.417820 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260928 02:11:30.419329 13973 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.6.235.124" port: 33005 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 02:11:30 dist-test-slave-86gv krb5kdc[13212](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561490, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.235.124@KRBTEST.COM for oryx/127.6.235.124@KRBTEST.COM
I20260928 02:11:30.431054 13978 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:11:30.419967 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.235.124:33005 (local address 127.0.0.1:41450)
0928 02:11:30.420114 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0928 02:11:30.420128 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 02:11:30.420203 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:11:30.420486 (+ 283us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:11:30.420489 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:11:30.420802 (+ 313us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:11:30.420988 (+ 186us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:11:30.420996 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:11:30.421925 (+ 929us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:11:30.421928 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:11:30.422736 (+ 808us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:11:30.422745 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:11:30.423374 (+ 629us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:11:30.423378 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:11:30.423491 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:11:30.424068 (+ 577us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:11:30.424085 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:11:30.426142 (+ 2057us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:11:30.429038 (+ 2896us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:11:30.429041 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:11:30.429044 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:11:30.429426 (+ 382us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:11:30.429776 (+ 350us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:11:30.429779 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:11:30.429781 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:11:30.429916 (+ 135us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:11:30.430343 (+ 427us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:11:30.430346 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:11:30.430466 (+ 120us) client_negotiation.cc:769] Sending connection context
0928 02:11:30.430573 (+ 107us) client_negotiation.cc:240] Negotiation successful
0928 02:11:30.430768 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":60}
I20260928 02:11:30.431174 13976 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:11:30.420124 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41450 (local address 127.6.235.124:33005)
0928 02:11:30.420274 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0928 02:11:30.420278 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:11:30.420292 (+ 14us) server_negotiation.cc:407] Connection header received
0928 02:11:30.420328 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:11:30.420332 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:11:30.420381 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:11:30.420456 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:11:30.421111 (+ 655us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:11:30.421818 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:11:30.422848 (+ 1030us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:11:30.423249 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:11:30.423337 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:11:30.426324 (+ 2987us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:11:30.426328 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:11:30.426331 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:11:30.426359 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:11:30.428861 (+ 2502us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:11:30.429530 (+ 669us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:11:30.429534 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:11:30.429537 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:11:30.429656 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:11:30.430005 (+ 349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:11:30.430008 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:11:30.430010 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:11:30.430249 (+ 239us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:11:30.430352 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0928 02:11:30.430652 (+ 300us) server_negotiation.cc:299] Negotiation successful
0928 02:11:30.430882 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":31,"server-negotiator.queue_time_us":50}
I20260928 02:11:30.435982 13973 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 254f36e26f8d4ddf823cf81c7a5c2986: Bootstrap starting.
I20260928 02:11:30.439599 13973 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 254f36e26f8d4ddf823cf81c7a5c2986: Neither blocks nor log segments found. Creating new log.
I20260928 02:11:30.440701 13973 log.cc:824] T 00000000000000000000000000000000 P 254f36e26f8d4ddf823cf81c7a5c2986: Log is configured to *not* fsync() on all Append() calls
I20260928 02:11:30.443544 13973 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 254f36e26f8d4ddf823cf81c7a5c2986: No bootstrap required, opened a new log
I20260928 02:11:30.451426 13973 raft_consensus.cc:348] T 00000000000000000000000000000000 P 254f36e26f8d4ddf823cf81c7a5c2986 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "11adc61886584956adfdac31047373c1" member_type: VOTER last_known_addr { host: "127.6.235.126" port: 34853 } } peers { permanent_uuid: "bafe67917b9a4b40a60409c753bf4b96" member_type: VOTER last_known_addr { host: "127.6.235.125" port: 41279 } } peers { permanent_uuid: "254f36e26f8d4ddf823cf81c7a5c2986" member_type: VOTER last_known_addr { host: "127.6.235.124" port: 33005 } }
I20260928 02:11:30.451776 13973 raft_consensus.cc:374] T 00000000000000000000000000000000 P 254f36e26f8d4ddf823cf81c7a5c2986 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:11:30.451865 13973 raft_consensus.cc:729] T 00000000000000000000000000000000 P 254f36e26f8d4ddf823cf81c7a5c2986 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 254f36e26f8d4ddf823cf81c7a5c2986, State: Initialized, Role: FOLLOWER
I20260928 02:11:30.452455 13973 consensus_queue.cc:260] T 00000000000000000000000000000000 P 254f36e26f8d4ddf823cf81c7a5c2986 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "11adc61886584956adfdac31047373c1" member_type: VOTER last_known_addr { host: "127.6.235.126" port: 34853 } } peers { permanent_uuid: "bafe67917b9a4b40a60409c753bf4b96" member_type: VOTER last_known_addr { host: "127.6.235.125" port: 41279 } } peers { permanent_uuid: "254f36e26f8d4ddf823cf81c7a5c2986" member_type: VOTER last_known_addr { host: "127.6.235.124" port: 33005 } }
I20260928 02:11:30.453440 13982 sys_catalog.cc:455] T 00000000000000000000000000000000 P 254f36e26f8d4ddf823cf81c7a5c2986 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "11adc61886584956adfdac31047373c1" member_type: VOTER last_known_addr { host: "127.6.235.126" port: 34853 } } peers { permanent_uuid: "bafe67917b9a4b40a60409c753bf4b96" member_type: VOTER last_known_addr { host: "127.6.235.125" port: 41279 } } peers { permanent_uuid: "254f36e26f8d4ddf823cf81c7a5c2986" member_type: VOTER last_known_addr { host: "127.6.235.124" port: 33005 } } }
I20260928 02:11:30.453785 13982 sys_catalog.cc:458] T 00000000000000000000000000000000 P 254f36e26f8d4ddf823cf81c7a5c2986 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 02:11:30.454506 13973 sys_catalog.cc:564] T 00000000000000000000000000000000 P 254f36e26f8d4ddf823cf81c7a5c2986 [sys.catalog]: configured and running, proceeding with master startup.
W20260928 02:11:30.464378 13993 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 254f36e26f8d4ddf823cf81c7a5c2986: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 02:11:30.464522 13993 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 02:11:30.469980 13821 master_runner.cc:428] Detected that this master 254f36e26f8d4ddf823cf81c7a5c2986 is joining an existing cluster
I20260928 02:11:30.470125 13821 master_runner.cc:432] Initiating AddMaster RPC to add 127.6.235.124:33005
I20260928 02:11:30.493428 13791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:11:30.474809 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42110 (local address 127.6.235.126:34853)
0928 02:11:30.475022 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0928 02:11:30.475030 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 02:11:30.475670 (+ 640us) server_negotiation.cc:407] Connection header received
0928 02:11:30.476001 (+ 331us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:11:30.476010 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:11:30.476149 (+ 139us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:11:30.476307 (+ 158us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:11:30.477702 (+ 1395us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:11:30.478885 (+ 1183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:11:30.481263 (+ 2378us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:11:30.481937 (+ 674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:11:30.482049 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:11:30.485410 (+ 3361us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:11:30.485418 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:11:30.485423 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:11:30.485468 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:11:30.489562 (+ 4094us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:11:30.490800 (+ 1238us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:11:30.490807 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:11:30.490811 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:11:30.490980 (+ 169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:11:30.491560 (+ 580us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:11:30.491566 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:11:30.491569 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:11:30.491924 (+ 355us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:11:30.492062 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0928 02:11:30.492748 (+ 686us) server_negotiation.cc:299] Negotiation successful
0928 02:11:30.493035 (+ 287us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260928 02:11:30.494624 13999 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:11:30.474825 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.235.126:34853 (local address 127.0.0.1:42110)
0928 02:11:30.475542 (+ 717us) negotiation.cc:107] Waiting for socket to connect
0928 02:11:30.475562 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0928 02:11:30.475818 (+ 256us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:11:30.476343 (+ 525us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:11:30.476348 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:11:30.477115 (+ 767us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:11:30.477511 (+ 396us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:11:30.477524 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:11:30.479094 (+ 1570us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:11:30.479099 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:11:30.481069 (+ 1970us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:11:30.481086 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:11:30.482771 (+ 1685us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:11:30.482778 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:11:30.483017 (+ 239us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:11:30.483910 (+ 893us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:11:30.483936 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:11:30.485160 (+ 1224us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:11:30.489739 (+ 4579us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:11:30.489745 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:11:30.489750 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:11:30.490637 (+ 887us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:11:30.491148 (+ 511us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:11:30.491153 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:11:30.491157 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:11:30.491422 (+ 265us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:11:30.492148 (+ 726us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:11:30.492153 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:11:30.492405 (+ 252us) client_negotiation.cc:769] Sending connection context
0928 02:11:30.493709 (+ 1304us) client_negotiation.cc:240] Negotiation successful
0928 02:11:30.494062 (+ 353us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":557,"mutex_wait_us":120,"thread_start_us":397,"threads_started":1}
I20260928 02:11:30.495501 13980 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:11:30.474870 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43142 (local address 127.6.235.125:41279)
0928 02:11:30.475214 (+ 344us) server_negotiation.cc:206] Beginning negotiation
0928 02:11:30.475266 (+ 52us) server_negotiation.cc:399] Waiting for connection header
0928 02:11:30.477565 (+ 2299us) server_negotiation.cc:407] Connection header received
0928 02:11:30.477858 (+ 293us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:11:30.477864 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:11:30.477945 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:11:30.478018 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:11:30.479996 (+ 1978us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:11:30.481120 (+ 1124us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:11:30.483411 (+ 2291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:11:30.484394 (+ 983us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:11:30.484525 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:11:30.488218 (+ 3693us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:11:30.488226 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:11:30.488233 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:11:30.488280 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:11:30.491967 (+ 3687us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:11:30.492802 (+ 835us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:11:30.492807 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:11:30.492811 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:11:30.492991 (+ 180us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:11:30.493457 (+ 466us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:11:30.493462 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:11:30.493465 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:11:30.493953 (+ 488us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:11:30.494067 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0928 02:11:30.494842 (+ 775us) server_negotiation.cc:299] Negotiation successful
0928 02:11:30.495113 (+ 271us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":185}
I20260928 02:11:30.496856 14000 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:11:30.475590 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.235.125:41279 (local address 127.0.0.1:43142)
0928 02:11:30.477466 (+ 1876us) negotiation.cc:107] Waiting for socket to connect
0928 02:11:30.477493 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0928 02:11:30.477674 (+ 181us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:11:30.478225 (+ 551us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:11:30.478231 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:11:30.479132 (+ 901us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:11:30.479786 (+ 654us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:11:30.479798 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:11:30.481313 (+ 1515us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:11:30.481318 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:11:30.483174 (+ 1856us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:11:30.483197 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:11:30.484964 (+ 1767us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:11:30.484973 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:11:30.485239 (+ 266us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:11:30.486375 (+ 1136us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:11:30.486408 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:11:30.487998 (+ 1590us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:11:30.492127 (+ 4129us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:11:30.492133 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:11:30.492137 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:11:30.492644 (+ 507us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:11:30.493158 (+ 514us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:11:30.493162 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:11:30.493166 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:11:30.493331 (+ 165us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:11:30.494155 (+ 824us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:11:30.494160 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:11:30.494467 (+ 307us) client_negotiation.cc:769] Sending connection context
0928 02:11:30.495948 (+ 1481us) client_negotiation.cc:240] Negotiation successful
0928 02:11:30.496315 (+ 367us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1720,"mutex_wait_us":108,"thread_start_us":1497,"threads_started":1}
I20260928 02:11:30.500427 7085 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskTlaUuc/build/asan/bin/kudu pid 13318
I20260928 02:11:30.504311 13976 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:11:30.477817 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41458 (local address 127.6.235.124:33005)
0928 02:11:30.477967 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0928 02:11:30.477973 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 02:11:30.487528 (+ 9555us) server_negotiation.cc:407] Connection header received
0928 02:11:30.487751 (+ 223us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:11:30.487758 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:11:30.487847 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:11:30.487954 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:11:30.489417 (+ 1463us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:11:30.490545 (+ 1128us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:11:30.492577 (+ 2032us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:11:30.493318 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:11:30.493437 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:11:30.496578 (+ 3141us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:11:30.496586 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:11:30.496590 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:11:30.496624 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:11:30.501740 (+ 5116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:11:30.502359 (+ 619us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:11:30.502367 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:11:30.502371 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:11:30.502481 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:11:30.502827 (+ 346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:11:30.502831 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:11:30.502835 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:11:30.503121 (+ 286us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:11:30.503272 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0928 02:11:30.503626 (+ 354us) server_negotiation.cc:299] Negotiation successful
0928 02:11:30.503955 (+ 329us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":137,"server-negotiator.queue_time_us":59}
I20260928 02:11:30.504369 14001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:11:30.478073 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.235.124:33005 (local address 127.0.0.1:41458)
0928 02:11:30.487399 (+ 9326us) negotiation.cc:107] Waiting for socket to connect
0928 02:11:30.487421 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0928 02:11:30.487581 (+ 160us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:11:30.488113 (+ 532us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:11:30.488121 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:11:30.488779 (+ 658us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:11:30.489250 (+ 471us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:11:30.489266 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:11:30.490766 (+ 1500us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:11:30.490772 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:11:30.492425 (+ 1653us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:11:30.492439 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:11:30.493963 (+ 1524us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:11:30.493970 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:11:30.494149 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:11:30.494908 (+ 759us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:11:30.494969 (+ 61us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:11:30.496361 (+ 1392us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:11:30.501865 (+ 5504us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:11:30.501873 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:11:30.501877 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:11:30.502245 (+ 368us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:11:30.502586 (+ 341us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:11:30.502593 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:11:30.502597 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:11:30.502718 (+ 121us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:11:30.503254 (+ 536us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:11:30.503258 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:11:30.503401 (+ 143us) client_negotiation.cc:769] Sending connection context
0928 02:11:30.503551 (+ 150us) client_negotiation.cc:240] Negotiation successful
0928 02:11:30.503829 (+ 278us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9124,"mutex_wait_us":30,"thread_start_us":8961,"threads_started":1}
Sep 28 02:11:30 dist-test-slave-86gv krb5kdc[13212](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561487, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.235.126@KRBTEST.COM for oryx/127.6.235.124@KRBTEST.COM
I20260928 02:11:30.522707 13979 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:11:30.500402 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.235.124:33005 (local address 127.0.0.1:41460)
0928 02:11:30.500724 (+ 322us) negotiation.cc:107] Waiting for socket to connect
0928 02:11:30.500742 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0928 02:11:30.500842 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:11:30.505529 (+ 4687us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:11:30.505536 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:11:30.506063 (+ 527us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:11:30.506356 (+ 293us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:11:30.506370 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:11:30.507741 (+ 1371us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:11:30.507746 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:11:30.508797 (+ 1051us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:11:30.508809 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:11:30.511461 (+ 2652us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:11:30.511468 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:11:30.511659 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:11:30.512452 (+ 793us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:11:30.512476 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:11:30.515570 (+ 3094us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:11:30.519882 (+ 4312us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:11:30.519889 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:11:30.519893 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:11:30.520403 (+ 510us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:11:30.520858 (+ 455us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:11:30.520863 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:11:30.520867 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:11:30.521076 (+ 209us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:11:30.521731 (+ 655us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:11:30.521736 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:11:30.521940 (+ 204us) client_negotiation.cc:769] Sending connection context
0928 02:11:30.522080 (+ 140us) client_negotiation.cc:240] Negotiation successful
0928 02:11:30.522371 (+ 291us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":207}
I20260928 02:11:30.523262 13976 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:11:30.501051 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41460 (local address 127.6.235.124:33005)
0928 02:11:30.505084 (+ 4033us) server_negotiation.cc:206] Beginning negotiation
0928 02:11:30.505096 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0928 02:11:30.505113 (+ 17us) server_negotiation.cc:407] Connection header received
0928 02:11:30.505200 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:11:30.505205 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:11:30.505335 (+ 130us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:11:30.506551 (+ 1216us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:11:30.506739 (+ 188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:11:30.507594 (+ 855us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:11:30.508971 (+ 1377us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:11:30.511262 (+ 2291us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:11:30.515795 (+ 4533us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:11:30.516035 (+ 240us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:11:30.516041 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:11:30.516046 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:11:30.516077 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:11:30.519720 (+ 3643us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:11:30.520549 (+ 829us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:11:30.520555 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:11:30.520559 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:11:30.520714 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:11:30.521213 (+ 499us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:11:30.521218 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:11:30.521222 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:11:30.521581 (+ 359us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:11:30.521749 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0928 02:11:30.522227 (+ 478us) server_negotiation.cc:299] Negotiation successful
0928 02:11:30.522471 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3892,"thread_start_us":15527,"threads_started":1}
I20260928 02:11:30.525080 13257 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.6.235.124:33005: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "254f36e26f8d4ddf823cf81c7a5c2986"
member_type: NON_VOTER
last_known_addr {
host: "127.6.235.124"
port: 33005
}
attrs {
promote: true
}
}
dest_uuid: "11adc61886584956adfdac31047373c1"
cas_config_opid_index: 5
I20260928 02:11:30.525979 13257 consensus_queue.cc:237] T 00000000000000000000000000000000 P 11adc61886584956adfdac31047373c1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "11adc61886584956adfdac31047373c1" member_type: VOTER last_known_addr { host: "127.6.235.126" port: 34853 } } peers { permanent_uuid: "bafe67917b9a4b40a60409c753bf4b96" member_type: NON_VOTER last_known_addr { host: "127.6.235.125" port: 41279 } attrs { promote: true } } peers { permanent_uuid: "254f36e26f8d4ddf823cf81c7a5c2986" member_type: NON_VOTER last_known_addr { host: "127.6.235.124" port: 33005 } attrs { promote: true } }
Sep 28 02:11:30 dist-test-slave-86gv krb5kdc[13212](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561486, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.6.235.65@KRBTEST.COM
I20260928 02:11:30.527938 13977 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 11adc61886584956adfdac31047373c1 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 254f36e26f8d4ddf823cf81c7a5c2986 (127.6.235.124) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "11adc61886584956adfdac31047373c1" member_type: VOTER last_known_addr { host: "127.6.235.126" port: 34853 } } peers { permanent_uuid: "bafe67917b9a4b40a60409c753bf4b96" member_type: NON_VOTER last_known_addr { host: "127.6.235.125" port: 41279 } attrs { promote: true } } peers { permanent_uuid: "254f36e26f8d4ddf823cf81c7a5c2986" member_type: NON_VOTER last_known_addr { host: "127.6.235.124" port: 33005 } attrs { promote: true } } }
I20260928 02:11:30.532073 13977 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.6.235.124:33005
I20260928 02:11:30.534824 13867 raft_consensus.cc:1261] T 00000000000000000000000000000000 P bafe67917b9a4b40a60409c753bf4b96 [term 1 LEARNER]: Refusing update from remote peer 11adc61886584956adfdac31047373c1: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260928 02:11:30.535205 13821 master.cc:537] Master@127.6.235.124:33005 shutting down...
I20260928 02:11:30.544819 14003 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:11:30.501440 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56836 (local address 127.6.235.65:42113)
0928 02:11:30.515703 (+ 14263us) server_negotiation.cc:206] Beginning negotiation
0928 02:11:30.515717 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0928 02:11:30.515763 (+ 46us) server_negotiation.cc:407] Connection header received
0928 02:11:30.515858 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:11:30.515871 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:11:30.516513 (+ 642us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:11:30.516814 (+ 301us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:11:30.518465 (+ 1651us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:11:30.520102 (+ 1637us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:11:30.521798 (+ 1696us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:11:30.522794 (+ 996us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:11:30.523507 (+ 713us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:11:30.529143 (+ 5636us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:11:30.529174 (+ 31us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:11:30.529195 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:11:30.529245 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:11:30.533587 (+ 4342us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:11:30.535022 (+ 1435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:11:30.535033 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:11:30.535036 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:11:30.535219 (+ 183us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:11:30.540053 (+ 4834us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:11:30.540062 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:11:30.540068 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:11:30.540849 (+ 781us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:11:30.543280 (+ 2431us) server_negotiation.cc:1035] Waiting for connection context
0928 02:11:30.543495 (+ 215us) server_negotiation.cc:299] Negotiation successful
0928 02:11:30.543853 (+ 358us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13972,"thread_start_us":14763,"threads_started":1}
I20260928 02:11:30.544778 13977 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 11adc61886584956adfdac31047373c1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bafe67917b9a4b40a60409c753bf4b96" member_type: NON_VOTER last_known_addr { host: "127.6.235.125" port: 41279 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260928 02:11:30.546895 13441 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 02:11:30.549279 14004 sys_catalog.cc:455] T 00000000000000000000000000000000 P 11adc61886584956adfdac31047373c1 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "11adc61886584956adfdac31047373c1" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "11adc61886584956adfdac31047373c1" member_type: VOTER last_known_addr { host: "127.6.235.126" port: 34853 } } peers { permanent_uuid: "bafe67917b9a4b40a60409c753bf4b96" member_type: NON_VOTER last_known_addr { host: "127.6.235.125" port: 41279 } attrs { promote: true } } peers { permanent_uuid: "254f36e26f8d4ddf823cf81c7a5c2986" member_type: NON_VOTER last_known_addr { host: "127.6.235.124" port: 33005 } attrs { promote: true } } }
I20260928 02:11:30.550293 14004 sys_catalog.cc:458] T 00000000000000000000000000000000 P 11adc61886584956adfdac31047373c1 [sys.catalog]: This master's current role is: LEADER
W20260928 02:11:30.556694 13244 consensus_peers.cc:487] T 00000000000000000000000000000000 P 11adc61886584956adfdac31047373c1 -> Peer 254f36e26f8d4ddf823cf81c7a5c2986 (127.6.235.124:33005): Couldn't send request to peer 254f36e26f8d4ddf823cf81c7a5c2986. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260928 02:11:30.564421 13821 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 254f36e26f8d4ddf823cf81c7a5c2986 [term 0 FOLLOWER]: Raft consensus shutting down.
I20260928 02:11:30.565023 13821 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 254f36e26f8d4ddf823cf81c7a5c2986 [term 0 FOLLOWER]: Raft consensus is shut down!
I20260928 02:11:30.565344 13821 tablet_replica.cc:330] T 00000000000000000000000000000000 P 254f36e26f8d4ddf823cf81c7a5c2986: stopping tablet replica
I20260928 02:11:30.575743 13821 master.cc:552] Master@127.6.235.124:33005 shutdown complete.
I20260928 02:11:30.575935 13821 master_runner.cc:305] Clearing existing system tablet
I20260928 02:11:30.576453 13821 fs_manager.cc:359] Metadata directory not provided
I20260928 02:11:30.576663 13821 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:11:30.578459 13867 raft_consensus.cc:2932] T 00000000000000000000000000000000 P bafe67917b9a4b40a60409c753bf4b96 [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 254f36e26f8d4ddf823cf81c7a5c2986 (127.6.235.124) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "11adc61886584956adfdac31047373c1" member_type: VOTER last_known_addr { host: "127.6.235.126" port: 34853 } } peers { permanent_uuid: "bafe67917b9a4b40a60409c753bf4b96" member_type: NON_VOTER last_known_addr { host: "127.6.235.125" port: 41279 } attrs { promote: true } } peers { permanent_uuid: "254f36e26f8d4ddf823cf81c7a5c2986" member_type: NON_VOTER last_known_addr { host: "127.6.235.124" port: 33005 } attrs { promote: true } } }
I20260928 02:11:30.579896 13821 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.001s
I20260928 02:11:30.582593 13895 sys_catalog.cc:455] T 00000000000000000000000000000000 P bafe67917b9a4b40a60409c753bf4b96 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "11adc61886584956adfdac31047373c1" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "11adc61886584956adfdac31047373c1" member_type: VOTER last_known_addr { host: "127.6.235.126" port: 34853 } } peers { permanent_uuid: "bafe67917b9a4b40a60409c753bf4b96" member_type: NON_VOTER last_known_addr { host: "127.6.235.125" port: 41279 } attrs { promote: true } } peers { permanent_uuid: "254f36e26f8d4ddf823cf81c7a5c2986" member_type: NON_VOTER last_known_addr { host: "127.6.235.124" port: 33005 } attrs { promote: true } } }
I20260928 02:11:30.582895 13895 sys_catalog.cc:458] T 00000000000000000000000000000000 P bafe67917b9a4b40a60409c753bf4b96 [sys.catalog]: This master's current role is: LEARNER
I20260928 02:11:30.583961 14004 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 11adc61886584956adfdac31047373c1: attempting to promote NON_VOTER bafe67917b9a4b40a60409c753bf4b96 to VOTER
I20260928 02:11:30.584697 14004 consensus_queue.cc:237] T 00000000000000000000000000000000 P 11adc61886584956adfdac31047373c1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "11adc61886584956adfdac31047373c1" member_type: VOTER last_known_addr { host: "127.6.235.126" port: 34853 } } peers { permanent_uuid: "bafe67917b9a4b40a60409c753bf4b96" member_type: VOTER last_known_addr { host: "127.6.235.125" port: 41279 } attrs { promote: false } } peers { permanent_uuid: "254f36e26f8d4ddf823cf81c7a5c2986" member_type: NON_VOTER last_known_addr { host: "127.6.235.124" port: 33005 } attrs { promote: true } }
I20260928 02:11:30.586839 13979 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:11:30.586324 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.235.124:33005 (local address 127.0.0.1:41462)
0928 02:11:30.586495 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0928 02:11:30.586635 (+ 140us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.235.124:33005: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
I20260928 02:11:30.587703 14011 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:11:30.588056 13821 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790561432251112-7085-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:11:30.588260 13979 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:11:30.587706 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.235.124:33005 (local address 127.0.0.1:41476)
0928 02:11:30.588003 (+ 297us) negotiation.cc:107] Waiting for socket to connect
0928 02:11:30.588108 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.235.124:33005: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":178}
I20260928 02:11:30.588330 13821 fs_manager.cc:725] Time spent opening block manager: real 0.007s user 0.004s sys 0.004s
I20260928 02:11:30.588428 13821 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790561432251112-7085-0/minicluster-data/master-2/data,/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790561432251112-7085-0/minicluster-data/master-2/wal
uuid: "254f36e26f8d4ddf823cf81c7a5c2986"
format_stamp: "Formatted at 2026-09-28 02:11:30 on dist-test-slave-86gv"
server_key: "64ad8e8c867224fbcc7569514b56e3a1"
server_key_iv: "120dfcb6e992c2672c0f57db7f35d68d"
server_key_version: "encryptionkey@0"
W20260928 02:11:30.588762 13244 consensus_peers.cc:487] T 00000000000000000000000000000000 P 11adc61886584956adfdac31047373c1 -> Peer 254f36e26f8d4ddf823cf81c7a5c2986 (127.6.235.124:33005): Couldn't send request to peer 254f36e26f8d4ddf823cf81c7a5c2986. Status: Network error: Client connection negotiation failed: client connection to 127.6.235.124:33005: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260928 02:11:30.592973 13867 raft_consensus.cc:1261] T 00000000000000000000000000000000 P bafe67917b9a4b40a60409c753bf4b96 [term 1 LEARNER]: Refusing update from remote peer 11adc61886584956adfdac31047373c1: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20260928 02:11:30.594147 13821 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 254f36e26f8d4ddf823cf81c7a5c2986: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 02:11:30.596578 14005 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 11adc61886584956adfdac31047373c1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bafe67917b9a4b40a60409c753bf4b96" member_type: VOTER last_known_addr { host: "127.6.235.125" port: 41279 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260928 02:11:30.599459 14004 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 11adc61886584956adfdac31047373c1 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, bafe67917b9a4b40a60409c753bf4b96 (127.6.235.125) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "11adc61886584956adfdac31047373c1" member_type: VOTER last_known_addr { host: "127.6.235.126" port: 34853 } } peers { permanent_uuid: "bafe67917b9a4b40a60409c753bf4b96" member_type: VOTER last_known_addr { host: "127.6.235.125" port: 41279 } attrs { promote: false } } peers { permanent_uuid: "254f36e26f8d4ddf823cf81c7a5c2986" member_type: NON_VOTER last_known_addr { host: "127.6.235.124" port: 33005 } attrs { promote: true } } }
I20260928 02:11:30.600211 13867 raft_consensus.cc:2932] T 00000000000000000000000000000000 P bafe67917b9a4b40a60409c753bf4b96 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, bafe67917b9a4b40a60409c753bf4b96 (127.6.235.125) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "11adc61886584956adfdac31047373c1" member_type: VOTER last_known_addr { host: "127.6.235.126" port: 34853 } } peers { permanent_uuid: "bafe67917b9a4b40a60409c753bf4b96" member_type: VOTER last_known_addr { host: "127.6.235.125" port: 41279 } attrs { promote: false } } peers { permanent_uuid: "254f36e26f8d4ddf823cf81c7a5c2986" member_type: NON_VOTER last_known_addr { host: "127.6.235.124" port: 33005 } attrs { promote: true } } }
I20260928 02:11:30.602241 13821 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 254f36e26f8d4ddf823cf81c7a5c2986: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260928 02:11:30.602396 13821 log.cc:1196] T 00000000000000000000000000000000 P 254f36e26f8d4ddf823cf81c7a5c2986: Deleting WAL directory at /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790561432251112-7085-0/minicluster-data/master-2/wal/wals/00000000000000000000000000000000
I20260928 02:11:30.602895 13821 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 254f36e26f8d4ddf823cf81c7a5c2986: Deleting consensus metadata
I20260928 02:11:30.603163 13821 master_runner.cc:315] Copying system tablet from 127.6.235.126:34853
I20260928 02:11:30.603530 13977 sys_catalog.cc:455] T 00000000000000000000000000000000 P 11adc61886584956adfdac31047373c1 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "11adc61886584956adfdac31047373c1" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "11adc61886584956adfdac31047373c1" member_type: VOTER last_known_addr { host: "127.6.235.126" port: 34853 } } peers { permanent_uuid: "bafe67917b9a4b40a60409c753bf4b96" member_type: VOTER last_known_addr { host: "127.6.235.125" port: 41279 } attrs { promote: false } } peers { permanent_uuid: "254f36e26f8d4ddf823cf81c7a5c2986" member_type: NON_VOTER last_known_addr { host: "127.6.235.124" port: 33005 } attrs { promote: true } } }
I20260928 02:11:30.603878 13977 sys_catalog.cc:458] T 00000000000000000000000000000000 P 11adc61886584956adfdac31047373c1 [sys.catalog]: This master's current role is: LEADER
I20260928 02:11:30.604189 14004 sys_catalog.cc:455] T 00000000000000000000000000000000 P 11adc61886584956adfdac31047373c1 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "11adc61886584956adfdac31047373c1" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "11adc61886584956adfdac31047373c1" member_type: VOTER last_known_addr { host: "127.6.235.126" port: 34853 } } peers { permanent_uuid: "bafe67917b9a4b40a60409c753bf4b96" member_type: VOTER last_known_addr { host: "127.6.235.125" port: 41279 } attrs { promote: false } } peers { permanent_uuid: "254f36e26f8d4ddf823cf81c7a5c2986" member_type: NON_VOTER last_known_addr { host: "127.6.235.124" port: 33005 } attrs { promote: true } } }
I20260928 02:11:30.604420 14004 sys_catalog.cc:458] T 00000000000000000000000000000000 P 11adc61886584956adfdac31047373c1 [sys.catalog]: This master's current role is: LEADER
I20260928 02:11:30.604271 13895 sys_catalog.cc:455] T 00000000000000000000000000000000 P bafe67917b9a4b40a60409c753bf4b96 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "11adc61886584956adfdac31047373c1" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "11adc61886584956adfdac31047373c1" member_type: VOTER last_known_addr { host: "127.6.235.126" port: 34853 } } peers { permanent_uuid: "bafe67917b9a4b40a60409c753bf4b96" member_type: VOTER last_known_addr { host: "127.6.235.125" port: 41279 } attrs { promote: false } } peers { permanent_uuid: "254f36e26f8d4ddf823cf81c7a5c2986" member_type: NON_VOTER last_known_addr { host: "127.6.235.124" port: 33005 } attrs { promote: true } } }
I20260928 02:11:30.604519 13895 sys_catalog.cc:458] T 00000000000000000000000000000000 P bafe67917b9a4b40a60409c753bf4b96 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 02:11:30.607564 13821 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 254f36e26f8d4ddf823cf81c7a5c2986: tablet copy: Beginning tablet copy session from remote peer at address 127.6.235.126:34853
I20260928 02:11:30.624313 13791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:11:30.608412 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42126 (local address 127.6.235.126:34853)
0928 02:11:30.608695 (+ 283us) server_negotiation.cc:206] Beginning negotiation
0928 02:11:30.608702 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 02:11:30.609467 (+ 765us) server_negotiation.cc:407] Connection header received
0928 02:11:30.609773 (+ 306us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:11:30.609779 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:11:30.609856 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:11:30.609954 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:11:30.611065 (+ 1111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:11:30.612156 (+ 1091us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:11:30.614256 (+ 2100us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:11:30.614934 (+ 678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:11:30.615059 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:11:30.617605 (+ 2546us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:11:30.617612 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:11:30.617617 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:11:30.617672 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:11:30.621214 (+ 3542us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:11:30.622047 (+ 833us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:11:30.622053 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:11:30.622057 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:11:30.622203 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:11:30.622660 (+ 457us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:11:30.622666 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:11:30.622670 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:11:30.623073 (+ 403us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:11:30.623220 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0928 02:11:30.623712 (+ 492us) server_negotiation.cc:299] Negotiation successful
0928 02:11:30.623939 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":144}
I20260928 02:11:30.624446 14017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:11:30.608408 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.235.126:34853 (local address 127.0.0.1:42126)
0928 02:11:30.609334 (+ 926us) negotiation.cc:107] Waiting for socket to connect
0928 02:11:30.609362 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0928 02:11:30.609518 (+ 156us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:11:30.610006 (+ 488us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:11:30.610019 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:11:30.610631 (+ 612us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:11:30.610892 (+ 261us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:11:30.610905 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:11:30.612327 (+ 1422us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:11:30.612334 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:11:30.614098 (+ 1764us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:11:30.614114 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:11:30.615121 (+ 1007us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:11:30.615128 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:11:30.615315 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:11:30.616132 (+ 817us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:11:30.616158 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:11:30.617388 (+ 1230us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:11:30.621361 (+ 3973us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:11:30.621367 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:11:30.621372 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:11:30.621888 (+ 516us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:11:30.622326 (+ 438us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:11:30.622332 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:11:30.622337 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:11:30.622529 (+ 192us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:11:30.623216 (+ 687us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:11:30.623222 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:11:30.623405 (+ 183us) client_negotiation.cc:769] Sending connection context
0928 02:11:30.623578 (+ 173us) client_negotiation.cc:240] Negotiation successful
0928 02:11:30.623936 (+ 358us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":763,"thread_start_us":645,"threads_started":1}
I20260928 02:11:30.625355 13277 tablet_copy_service.cc:140] P 11adc61886584956adfdac31047373c1: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 254f36e26f8d4ddf823cf81c7a5c2986 ({username='oryx', principal='oryx/127.6.235.124@KRBTEST.COM'} at 127.0.0.1:42126)
I20260928 02:11:30.625536 13277 tablet_copy_service.cc:161] P 11adc61886584956adfdac31047373c1: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 254f36e26f8d4ddf823cf81c7a5c2986 at {username='oryx', principal='oryx/127.6.235.124@KRBTEST.COM'} at 127.0.0.1:42126: session id = 254f36e26f8d4ddf823cf81c7a5c2986-00000000000000000000000000000000
I20260928 02:11:30.629218 13277 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 11adc61886584956adfdac31047373c1: Tablet Copy: opened 0 blocks and 1 log segments
I20260928 02:11:30.631493 13821 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260928 02:11:30.639357 13821 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 254f36e26f8d4ddf823cf81c7a5c2986: tablet copy: Starting download of 0 data blocks...
I20260928 02:11:30.639688 13821 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 254f36e26f8d4ddf823cf81c7a5c2986: tablet copy: Starting download of 1 WAL segments...
I20260928 02:11:30.642349 13821 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 254f36e26f8d4ddf823cf81c7a5c2986: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260928 02:11:30.651003 13277 tablet_copy_service.cc:342] P 11adc61886584956adfdac31047373c1: Request end of tablet copy session 254f36e26f8d4ddf823cf81c7a5c2986-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.6.235.124@KRBTEST.COM'} at 127.0.0.1:42126
I20260928 02:11:30.651202 13277 tablet_copy_service.cc:433] P 11adc61886584956adfdac31047373c1: ending tablet copy session 254f36e26f8d4ddf823cf81c7a5c2986-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 254f36e26f8d4ddf823cf81c7a5c2986
I20260928 02:11:30.662889 13821 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:11:30.666095 14018 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:11:30.666363 14019 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:11:30.675339 13821 server_base.cc:1034] running on GCE node
W20260928 02:11:30.675554 14021 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:11:30.676074 13821 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:11:30.676793 13821 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:11:30.678020 13821 hybrid_clock.cc:648] HybridClock initialized: now 1790561490677935 us; error 40 us; skew 500 ppm
I20260928 02:11:30.678514 13821 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 02:11:30 dist-test-slave-86gv krb5kdc[13212](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561490, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.235.124@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:11:30.681006 13821 init.cc:377] Logged in from keytab as oryx/127.6.235.124@KRBTEST.COM (short username oryx)
I20260928 02:11:30.682479 13821 webserver.cc:460] Webserver started at http://127.6.235.124:37359/ using document root <none> and password file <none>
I20260928 02:11:30.682868 13821 fs_manager.cc:359] Metadata directory not provided
I20260928 02:11:30.683012 13821 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:11:30.686396 13821 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20260928 02:11:30.688464 14028 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:11:30.689021 13821 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.002s sys 0.000s
I20260928 02:11:30.689152 13821 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790561432251112-7085-0/minicluster-data/master-2/data,/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790561432251112-7085-0/minicluster-data/master-2/wal
uuid: "254f36e26f8d4ddf823cf81c7a5c2986"
format_stamp: "Formatted at 2026-09-28 02:11:30 on dist-test-slave-86gv"
server_key: "64ad8e8c867224fbcc7569514b56e3a1"
server_key_iv: "120dfcb6e992c2672c0f57db7f35d68d"
server_key_version: "encryptionkey@0"
I20260928 02:11:30.689350 13821 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790561432251112-7085-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790561432251112-7085-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790561432251112-7085-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:11:30.701586 13821 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:11:30.706378 13821 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:11:30.706997 7085 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTlaUuc/build/asan/bin/kudu with pid 13318
I20260928 02:11:30.731564 7085 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTlaUuc/build/asan/bin/kudu
/tmp/dist-test-taskTlaUuc/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790561432251112-7085-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790561432251112-7085-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790561432251112-7085-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790561432251112-7085-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.235.65:42113
--local_ip_for_outbound_sockets=127.6.235.65
--tserver_master_addrs=127.6.235.126:34853
--webserver_port=34129
--webserver_interface=127.6.235.65
--keytab_file=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790561432251112-7085-0/krb5kdc/oryx_127.6.235.65.keytab
--principal=oryx/127.6.235.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.235.84:44519
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.6.235.126:34853,127.6.235.125:41279
--tserver_master_addrs=127.6.235.126:34853,127.6.235.125:41279,127.6.235.124:33005 with env {KRB5CCNAME=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790561432251112-7085-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790561432251112-7085-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790561432251112-7085-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 02:11:30.753253 13821 rpc_server.cc:307] RPC server started. Bound to: 127.6.235.124:33005
I20260928 02:11:30.753307 14081 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.235.124:33005 every 8 connection(s)
I20260928 02:11:30.754612 13821 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790561432251112-7085-0/minicluster-data/master-2/data/info.pb
I20260928 02:11:30.762594 14082 sys_catalog.cc:263] Verifying existing consensus state
I20260928 02:11:30.767323 14082 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 254f36e26f8d4ddf823cf81c7a5c2986: Bootstrap starting.
I20260928 02:11:30.788101 14082 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 254f36e26f8d4ddf823cf81c7a5c2986: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 02:11:30.788705 14082 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 254f36e26f8d4ddf823cf81c7a5c2986: Bootstrap complete.
I20260928 02:11:30.789991 14082 raft_consensus.cc:348] T 00000000000000000000000000000000 P 254f36e26f8d4ddf823cf81c7a5c2986 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "11adc61886584956adfdac31047373c1" member_type: VOTER last_known_addr { host: "127.6.235.126" port: 34853 } } peers { permanent_uuid: "bafe67917b9a4b40a60409c753bf4b96" member_type: VOTER last_known_addr { host: "127.6.235.125" port: 41279 } attrs { promote: false } } peers { permanent_uuid: "254f36e26f8d4ddf823cf81c7a5c2986" member_type: NON_VOTER last_known_addr { host: "127.6.235.124" port: 33005 } attrs { promote: true } }
I20260928 02:11:30.790321 14082 raft_consensus.cc:729] T 00000000000000000000000000000000 P 254f36e26f8d4ddf823cf81c7a5c2986 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 254f36e26f8d4ddf823cf81c7a5c2986, State: Initialized, Role: LEARNER
I20260928 02:11:30.790594 14082 consensus_queue.cc:260] T 00000000000000000000000000000000 P 254f36e26f8d4ddf823cf81c7a5c2986 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "11adc61886584956adfdac31047373c1" member_type: VOTER last_known_addr { host: "127.6.235.126" port: 34853 } } peers { permanent_uuid: "bafe67917b9a4b40a60409c753bf4b96" member_type: VOTER last_known_addr { host: "127.6.235.125" port: 41279 } attrs { promote: false } } peers { permanent_uuid: "254f36e26f8d4ddf823cf81c7a5c2986" member_type: NON_VOTER last_known_addr { host: "127.6.235.124" port: 33005 } attrs { promote: true } }
I20260928 02:11:30.791501 14085 sys_catalog.cc:455] T 00000000000000000000000000000000 P 254f36e26f8d4ddf823cf81c7a5c2986 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "11adc61886584956adfdac31047373c1" member_type: VOTER last_known_addr { host: "127.6.235.126" port: 34853 } } peers { permanent_uuid: "bafe67917b9a4b40a60409c753bf4b96" member_type: VOTER last_known_addr { host: "127.6.235.125" port: 41279 } attrs { promote: false } } peers { permanent_uuid: "254f36e26f8d4ddf823cf81c7a5c2986" member_type: NON_VOTER last_known_addr { host: "127.6.235.124" port: 33005 } attrs { promote: true } } }
I20260928 02:11:30.791817 14082 sys_catalog.cc:564] T 00000000000000000000000000000000 P 254f36e26f8d4ddf823cf81c7a5c2986 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 02:11:30.792001 14085 sys_catalog.cc:458] T 00000000000000000000000000000000 P 254f36e26f8d4ddf823cf81c7a5c2986 [sys.catalog]: This master's current role is: LEARNER
I20260928 02:11:30.802297 14096 catalog_manager.cc:1259] Loaded cluster ID: 33ac180e627642b2a4850f9fab0c8be0
I20260928 02:11:30.802451 14096 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 254f36e26f8d4ddf823cf81c7a5c2986: loading cluster ID for follower catalog manager: success
I20260928 02:11:30.806733 14096 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 254f36e26f8d4ddf823cf81c7a5c2986: acquiring CA information for follower catalog manager: success
I20260928 02:11:30.809228 14096 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 254f36e26f8d4ddf823cf81c7a5c2986: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260928 02:11:31.111335 14070 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:11:31.111698 14070 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:11:31.111799 14070 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:11:31.119606 14070 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:11:31.119732 14070 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:11:31.119846 14070 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.235.65
I20260928 02:11:31.129936 14070 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.235.84:44519
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790561432251112-7085-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790561432251112-7085-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790561432251112-7085-0/krb5kdc/oryx_127.6.235.65.keytab
--principal=oryx/127.6.235.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.235.65:42113
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790561432251112-7085-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.6.235.65
--webserver_port=34129
--tserver_master_addrs=127.6.235.126:34853,127.6.235.125:41279,127.6.235.124:33005
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.235.65
--log_dir=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790561432251112-7085-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 00:51:59 UTC on ccycloud.quasar-axgkwn.root.comops.site
build id 4676464
ASAN enabled
I20260928 02:11:31.131348 14070 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:11:31.132766 14070 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:11:31.146910 14102 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:11:31.146871 14104 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:11:31.147526 14101 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:11:31.150575 14070 server_base.cc:1034] running on GCE node
I20260928 02:11:31.151430 14070 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:11:31.152578 14070 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:11:31.153806 14070 hybrid_clock.cc:648] HybridClock initialized: now 1790561491153746 us; error 49 us; skew 500 ppm
I20260928 02:11:31.154255 14070 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 02:11:31 dist-test-slave-86gv krb5kdc[13212](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561491, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.235.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:11:31.157838 14070 init.cc:377] Logged in from keytab as oryx/127.6.235.65@KRBTEST.COM (short username oryx)
I20260928 02:11:31.159349 14070 webserver.cc:460] Webserver started at http://127.6.235.65:34129/ using document root <none> and password file <none>
I20260928 02:11:31.160147 14070 fs_manager.cc:359] Metadata directory not provided
I20260928 02:11:31.160274 14070 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:11:31.166947 14070 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.004s sys 0.000s
I20260928 02:11:31.170020 14114 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:11:31.171602 14112 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:11:31.165143 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.235.124:33005 (local address 127.0.0.1:41484)
0928 02:11:31.166027 (+ 884us) negotiation.cc:107] Waiting for socket to connect
0928 02:11:31.166043 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 02:11:31.166203 (+ 160us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:11:31.166573 (+ 370us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:11:31.166578 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:11:31.166600 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:11:31.166829 (+ 229us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:11:31.166838 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:11:31.168113 (+ 1275us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:11:31.168119 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:11:31.169384 (+ 1265us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:11:31.169396 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:11:31.170872 (+ 1476us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:11:31.170879 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:11:31.171093 (+ 214us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:11:31.171122 (+ 29us) client_negotiation.cc:769] Sending connection context
0928 02:11:31.171213 (+ 91us) client_negotiation.cc:240] Negotiation successful
0928 02:11:31.171384 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":768,"thread_start_us":615,"threads_started":1}
I20260928 02:11:31.171743 14113 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:11:31.165305 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41484 (local address 127.6.235.124:33005)
0928 02:11:31.165905 (+ 600us) server_negotiation.cc:206] Beginning negotiation
0928 02:11:31.165912 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 02:11:31.166131 (+ 219us) server_negotiation.cc:407] Connection header received
0928 02:11:31.166344 (+ 213us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:11:31.166349 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:11:31.166438 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:11:31.166559 (+ 121us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:11:31.166965 (+ 406us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:11:31.167963 (+ 998us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:11:31.169571 (+ 1608us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:11:31.170684 (+ 1113us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:11:31.170840 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:11:31.171062 (+ 222us) server_negotiation.cc:1035] Waiting for connection context
0928 02:11:31.171253 (+ 191us) server_negotiation.cc:299] Negotiation successful
0928 02:11:31.171384 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":436,"thread_start_us":325,"threads_started":1}
I20260928 02:11:31.172096 14070 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20260928 02:11:31.172856 14055 raft_consensus.cc:1203] T 00000000000000000000000000000000 P 254f36e26f8d4ddf823cf81c7a5c2986 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20260928 02:11:31.172312 14070 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790561432251112-7085-0/minicluster-data/ts-0/data,/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790561432251112-7085-0/minicluster-data/ts-0/wal
uuid: "db7477740b844e9c8e8d14f8724f78d3"
format_stamp: "Formatted at 2026-09-28 02:11:27 on dist-test-slave-86gv"
server_key: "0f1b47083a67517a1f97fd57748b86f9"
server_key_iv: "8f217dfca3ef6ce3ddde88656bca0f10"
server_key_version: "encryptionkey@0"
I20260928 02:11:31.173517 14085 sys_catalog.cc:455] T 00000000000000000000000000000000 P 254f36e26f8d4ddf823cf81c7a5c2986 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 11adc61886584956adfdac31047373c1. Latest consensus state: current_term: 1 leader_uuid: "11adc61886584956adfdac31047373c1" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "11adc61886584956adfdac31047373c1" member_type: VOTER last_known_addr { host: "127.6.235.126" port: 34853 } } peers { permanent_uuid: "bafe67917b9a4b40a60409c753bf4b96" member_type: VOTER last_known_addr { host: "127.6.235.125" port: 41279 } attrs { promote: false } } peers { permanent_uuid: "254f36e26f8d4ddf823cf81c7a5c2986" member_type: NON_VOTER last_known_addr { host: "127.6.235.124" port: 33005 } attrs { promote: true } } }
I20260928 02:11:31.173799 14085 sys_catalog.cc:458] T 00000000000000000000000000000000 P 254f36e26f8d4ddf823cf81c7a5c2986 [sys.catalog]: This master's current role is: LEARNER
I20260928 02:11:31.173750 14070 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790561432251112-7085-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790561432251112-7085-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790561432251112-7085-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:11:31.199673 14070 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:11:31.204489 14070 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:11:31.204732 14070 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:11:31.205763 14070 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:11:31.207350 14070 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:11:31.207436 14070 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:11:31.207515 14070 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:11:31.207620 14070 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:11:31.241819 14070 rpc_server.cc:307] RPC server started. Bound to: 127.6.235.65:42113
I20260928 02:11:31.241945 14226 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.235.65:42113 every 8 connection(s)
I20260928 02:11:31.243491 14070 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790561432251112-7085-0/minicluster-data/ts-0/data/info.pb
I20260928 02:11:31.244103 7085 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTlaUuc/build/asan/bin/kudu as pid 14070
I20260928 02:11:31.244467 7085 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskTlaUuc/build/asan/bin/kudu pid 13452
Sep 28 02:11:31 dist-test-slave-86gv krb5kdc[13212](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561491, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.235.65@KRBTEST.COM for oryx/127.6.235.124@KRBTEST.COM
Sep 28 02:11:31 dist-test-slave-86gv krb5kdc[13212](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561486, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.6.235.66@KRBTEST.COM
I20260928 02:11:31.296396 14113 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:11:31.246617 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.235.65:46187 (local address 127.6.235.124:33005)
0928 02:11:31.246831 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0928 02:11:31.246837 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 02:11:31.248379 (+ 1542us) server_negotiation.cc:407] Connection header received
0928 02:11:31.260594 (+ 12215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:11:31.260606 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:11:31.260768 (+ 162us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:11:31.260873 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:11:31.264126 (+ 3253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:11:31.265684 (+ 1558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:11:31.269213 (+ 3529us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:11:31.270184 (+ 971us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:11:31.270332 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:11:31.285881 (+ 15549us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:11:31.285892 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:11:31.285899 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:11:31.285962 (+ 63us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:11:31.290715 (+ 4753us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:11:31.292464 (+ 1749us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:11:31.292473 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:11:31.292480 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:11:31.292680 (+ 200us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:11:31.293448 (+ 768us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:11:31.293456 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:11:31.293462 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:11:31.293925 (+ 463us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:11:31.294089 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0928 02:11:31.295549 (+ 1460us) server_negotiation.cc:299] Negotiation successful
0928 02:11:31.295894 (+ 345us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
Sep 28 02:11:31 dist-test-slave-86gv krb5kdc[13212](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561491, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.235.65@KRBTEST.COM for oryx/127.6.235.126@KRBTEST.COM
I20260928 02:11:31.299674 14231 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:11:31.247305 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.235.124:33005 (local address 127.6.235.65:46187)
0928 02:11:31.248174 (+ 869us) negotiation.cc:107] Waiting for socket to connect
0928 02:11:31.248228 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0928 02:11:31.258224 (+ 9996us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:11:31.261850 (+ 3626us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:11:31.261869 (+ 19us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:11:31.262965 (+ 1096us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:11:31.263896 (+ 931us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:11:31.263919 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:11:31.265978 (+ 2059us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:11:31.265987 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:11:31.268959 (+ 2972us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:11:31.268982 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:11:31.270883 (+ 1901us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:11:31.270893 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:11:31.271897 (+ 1004us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:11:31.275129 (+ 3232us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:11:31.275185 (+ 56us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:11:31.284986 (+ 9801us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:11:31.290947 (+ 5961us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:11:31.290962 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:11:31.290991 (+ 29us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:11:31.291912 (+ 921us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:11:31.292886 (+ 974us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:11:31.292912 (+ 26us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:11:31.292919 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:11:31.293276 (+ 357us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:11:31.294225 (+ 949us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:11:31.294242 (+ 17us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:11:31.294929 (+ 687us) client_negotiation.cc:769] Sending connection context
0928 02:11:31.296771 (+ 1842us) client_negotiation.cc:240] Negotiation successful
0928 02:11:31.297614 (+ 843us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":565,"mutex_wait_us":1,"thread_start_us":362,"threads_started":1}
I20260928 02:11:31.301944 14227 heartbeater.cc:347] Connected to a master server at 127.6.235.124:33005
I20260928 02:11:31.302310 14227 heartbeater.cc:464] Registering TS with master...
I20260928 02:11:31.303548 14227 heartbeater.cc:511] Master 127.6.235.124:33005 requested a full tablet report, sending...
Sep 28 02:11:31 dist-test-slave-86gv krb5kdc[13212](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561491, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.235.65@KRBTEST.COM for oryx/127.6.235.125@KRBTEST.COM
I20260928 02:11:31.305683 14230 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:11:31.245779 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44546 (local address 127.6.235.66:42855)
0928 02:11:31.271967 (+ 26188us) server_negotiation.cc:206] Beginning negotiation
0928 02:11:31.271977 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 02:11:31.272008 (+ 31us) server_negotiation.cc:407] Connection header received
0928 02:11:31.272073 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:11:31.272081 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:11:31.272402 (+ 321us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:11:31.272589 (+ 187us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:11:31.277034 (+ 4445us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:11:31.281040 (+ 4006us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:11:31.283759 (+ 2719us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:11:31.285116 (+ 1357us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:11:31.291279 (+ 6163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:11:31.291594 (+ 315us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:11:31.291612 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:11:31.291633 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:11:31.291678 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:11:31.295714 (+ 4036us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:11:31.297784 (+ 2070us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:11:31.297794 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:11:31.297797 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:11:31.297972 (+ 175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:11:31.298461 (+ 489us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:11:31.298466 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:11:31.298471 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:11:31.298908 (+ 437us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:11:31.303937 (+ 5029us) server_negotiation.cc:1035] Waiting for connection context
0928 02:11:31.304152 (+ 215us) server_negotiation.cc:299] Negotiation successful
0928 02:11:31.304566 (+ 414us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26002,"thread_start_us":27086,"threads_started":1}
I20260928 02:11:31.306381 14045 ts_manager.cc:194] Registered new tserver with Master: db7477740b844e9c8e8d14f8724f78d3 (127.6.235.65:42113)
I20260928 02:11:31.307214 13575 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 02:11:31.308750 14234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:11:31.251512 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.235.126:34853 (local address 127.6.235.65:56167)
0928 02:11:31.269055 (+ 17543us) negotiation.cc:107] Waiting for socket to connect
0928 02:11:31.269084 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0928 02:11:31.269282 (+ 198us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:11:31.270067 (+ 785us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:11:31.270075 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:11:31.270938 (+ 863us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:11:31.271541 (+ 603us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:11:31.271561 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:11:31.280305 (+ 8744us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:11:31.280315 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:11:31.283856 (+ 3541us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:11:31.283881 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:11:31.290414 (+ 6533us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:11:31.290420 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:11:31.290631 (+ 211us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:11:31.291389 (+ 758us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:11:31.291416 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:11:31.298711 (+ 7295us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:11:31.305911 (+ 7200us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:11:31.305916 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:11:31.305920 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:11:31.306514 (+ 594us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:11:31.307013 (+ 499us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:11:31.307018 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:11:31.307021 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:11:31.307189 (+ 168us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:11:31.307814 (+ 625us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:11:31.307819 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:11:31.307976 (+ 157us) client_negotiation.cc:769] Sending connection context
0928 02:11:31.308084 (+ 108us) client_negotiation.cc:240] Negotiation successful
0928 02:11:31.308304 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":17323,"thread_start_us":17018,"threads_started":1}
I20260928 02:11:31.309648 14233 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:11:31.251479 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.235.65:56167 (local address 127.6.235.126:34853)
0928 02:11:31.268260 (+ 16781us) server_negotiation.cc:206] Beginning negotiation
0928 02:11:31.268269 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 02:11:31.269429 (+ 1160us) server_negotiation.cc:407] Connection header received
0928 02:11:31.269517 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:11:31.269526 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:11:31.269634 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:11:31.269783 (+ 149us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:11:31.271794 (+ 2011us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:11:31.277746 (+ 5952us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:11:31.289119 (+ 11373us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:11:31.289962 (+ 843us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:11:31.290106 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:11:31.300193 (+ 10087us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:11:31.300202 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:11:31.300206 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:11:31.300257 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:11:31.305732 (+ 5475us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:11:31.306697 (+ 965us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:11:31.306701 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:11:31.306705 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:11:31.306866 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:11:31.307336 (+ 470us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:11:31.307340 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:11:31.307343 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:11:31.307675 (+ 332us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:11:31.309031 (+ 1356us) server_negotiation.cc:1035] Waiting for connection context
0928 02:11:31.309141 (+ 110us) server_negotiation.cc:299] Negotiation successful
0928 02:11:31.309326 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16572,"thread_start_us":16797,"threads_started":1}
I20260928 02:11:31.310707 14228 heartbeater.cc:347] Connected to a master server at 127.6.235.126:34853
I20260928 02:11:31.310875 14228 heartbeater.cc:464] Registering TS with master...
I20260928 02:11:31.311293 14228 heartbeater.cc:511] Master 127.6.235.126:34853 requested a full tablet report, sending...
I20260928 02:11:31.312512 13257 ts_manager.cc:194] Re-registered known tserver with Master: db7477740b844e9c8e8d14f8724f78d3 (127.6.235.65:42113)
I20260928 02:11:31.313680 13257 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.6.235.65@KRBTEST.COM'} at 127.6.235.65:56167
I20260928 02:11:31.315596 14237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:11:31.274109 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.235.65:48153 (local address 127.6.235.125:41279)
0928 02:11:31.292508 (+ 18399us) server_negotiation.cc:206] Beginning negotiation
0928 02:11:31.292516 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 02:11:31.292547 (+ 31us) server_negotiation.cc:407] Connection header received
0928 02:11:31.292613 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:11:31.292620 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:11:31.292825 (+ 205us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:11:31.292962 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:11:31.295016 (+ 2054us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:11:31.296939 (+ 1923us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:11:31.301082 (+ 4143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:11:31.301772 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:11:31.301929 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:11:31.307642 (+ 5713us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:11:31.307653 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:11:31.307661 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:11:31.307735 (+ 74us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:11:31.312293 (+ 4558us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:11:31.313227 (+ 934us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:11:31.313234 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:11:31.313239 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:11:31.313386 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:11:31.313840 (+ 454us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:11:31.313844 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:11:31.313848 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:11:31.314430 (+ 582us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:11:31.314565 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0928 02:11:31.314992 (+ 427us) server_negotiation.cc:299] Negotiation successful
0928 02:11:31.315199 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18247,"thread_start_us":18130,"threads_started":1}
I20260928 02:11:31.315625 14236 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:11:31.273934 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.235.125:41279 (local address 127.6.235.65:48153)
0928 02:11:31.286288 (+ 12354us) negotiation.cc:107] Waiting for socket to connect
0928 02:11:31.286309 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0928 02:11:31.286445 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:11:31.293206 (+ 6761us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:11:31.293211 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:11:31.293952 (+ 741us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:11:31.294839 (+ 887us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:11:31.294856 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:11:31.297101 (+ 2245us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:11:31.297106 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:11:31.299929 (+ 2823us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:11:31.299943 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:11:31.302040 (+ 2097us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:11:31.302051 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:11:31.302332 (+ 281us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:11:31.303007 (+ 675us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:11:31.303036 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:11:31.307327 (+ 4291us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:11:31.312485 (+ 5158us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:11:31.312492 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:11:31.312497 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:11:31.313076 (+ 579us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:11:31.313508 (+ 432us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:11:31.313513 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:11:31.313516 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:11:31.313685 (+ 169us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:11:31.314544 (+ 859us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:11:31.314548 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:11:31.314738 (+ 190us) client_negotiation.cc:769] Sending connection context
0928 02:11:31.314883 (+ 145us) client_negotiation.cc:240] Negotiation successful
0928 02:11:31.315198 (+ 315us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12176,"mutex_wait_us":2,"thread_start_us":12589,"threads_started":1}
I20260928 02:11:31.316979 14232 heartbeater.cc:347] Connected to a master server at 127.6.235.125:41279
I20260928 02:11:31.317147 14232 heartbeater.cc:464] Registering TS with master...
I20260928 02:11:31.317916 14232 heartbeater.cc:511] Master 127.6.235.125:41279 requested a full tablet report, sending...
I20260928 02:11:31.319837 13857 ts_manager.cc:194] Registered new tserver with Master: db7477740b844e9c8e8d14f8724f78d3 (127.6.235.65:42113)
I20260928 02:11:31.409818 7085 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTlaUuc/build/asan/bin/kudu with pid 13452
I20260928 02:11:31.427234 7085 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTlaUuc/build/asan/bin/kudu
/tmp/dist-test-taskTlaUuc/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790561432251112-7085-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790561432251112-7085-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790561432251112-7085-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790561432251112-7085-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.235.66:42855
--local_ip_for_outbound_sockets=127.6.235.66
--tserver_master_addrs=127.6.235.126:34853
--webserver_port=33027
--webserver_interface=127.6.235.66
--keytab_file=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790561432251112-7085-0/krb5kdc/oryx_127.6.235.66.keytab
--principal=oryx/127.6.235.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.235.84:44519
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.6.235.126:34853,127.6.235.125:41279
--tserver_master_addrs=127.6.235.126:34853,127.6.235.125:41279,127.6.235.124:33005 with env {KRB5CCNAME=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790561432251112-7085-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790561432251112-7085-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790561432251112-7085-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 02:11:31.677878 14110 sys_catalog.cc:455] T 00000000000000000000000000000000 P 11adc61886584956adfdac31047373c1 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "11adc61886584956adfdac31047373c1" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "11adc61886584956adfdac31047373c1" member_type: VOTER last_known_addr { host: "127.6.235.126" port: 34853 } } peers { permanent_uuid: "bafe67917b9a4b40a60409c753bf4b96" member_type: VOTER last_known_addr { host: "127.6.235.125" port: 41279 } attrs { promote: false } } peers { permanent_uuid: "254f36e26f8d4ddf823cf81c7a5c2986" member_type: NON_VOTER last_known_addr { host: "127.6.235.124" port: 33005 } attrs { promote: true } } }
I20260928 02:11:31.678275 14110 sys_catalog.cc:458] T 00000000000000000000000000000000 P 11adc61886584956adfdac31047373c1 [sys.catalog]: This master's current role is: LEADER
I20260928 02:11:31.679888 14241 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 11adc61886584956adfdac31047373c1: attempting to promote NON_VOTER 254f36e26f8d4ddf823cf81c7a5c2986 to VOTER
I20260928 02:11:31.680491 14241 consensus_queue.cc:237] T 00000000000000000000000000000000 P 11adc61886584956adfdac31047373c1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "11adc61886584956adfdac31047373c1" member_type: VOTER last_known_addr { host: "127.6.235.126" port: 34853 } } peers { permanent_uuid: "bafe67917b9a4b40a60409c753bf4b96" member_type: VOTER last_known_addr { host: "127.6.235.125" port: 41279 } attrs { promote: false } } peers { permanent_uuid: "254f36e26f8d4ddf823cf81c7a5c2986" member_type: VOTER last_known_addr { host: "127.6.235.124" port: 33005 } attrs { promote: false } }
I20260928 02:11:31.682615 13867 raft_consensus.cc:1261] T 00000000000000000000000000000000 P bafe67917b9a4b40a60409c753bf4b96 [term 1 FOLLOWER]: Refusing update from remote peer 11adc61886584956adfdac31047373c1: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20260928 02:11:31.682778 14055 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 254f36e26f8d4ddf823cf81c7a5c2986 [term 1 LEARNER]: Refusing update from remote peer 11adc61886584956adfdac31047373c1: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20260928 02:11:31.683274 14241 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 11adc61886584956adfdac31047373c1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bafe67917b9a4b40a60409c753bf4b96" member_type: VOTER last_known_addr { host: "127.6.235.125" port: 41279 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260928 02:11:31.683724 14110 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 11adc61886584956adfdac31047373c1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "254f36e26f8d4ddf823cf81c7a5c2986" member_type: VOTER last_known_addr { host: "127.6.235.124" port: 33005 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260928 02:11:31.688457 14241 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 11adc61886584956adfdac31047373c1 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 254f36e26f8d4ddf823cf81c7a5c2986 (127.6.235.124) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "11adc61886584956adfdac31047373c1" member_type: VOTER last_known_addr { host: "127.6.235.126" port: 34853 } } peers { permanent_uuid: "bafe67917b9a4b40a60409c753bf4b96" member_type: VOTER last_known_addr { host: "127.6.235.125" port: 41279 } attrs { promote: false } } peers { permanent_uuid: "254f36e26f8d4ddf823cf81c7a5c2986" member_type: VOTER last_known_addr { host: "127.6.235.124" port: 33005 } attrs { promote: false } } }
I20260928 02:11:31.689546 14055 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 254f36e26f8d4ddf823cf81c7a5c2986 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 254f36e26f8d4ddf823cf81c7a5c2986 (127.6.235.124) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "11adc61886584956adfdac31047373c1" member_type: VOTER last_known_addr { host: "127.6.235.126" port: 34853 } } peers { permanent_uuid: "bafe67917b9a4b40a60409c753bf4b96" member_type: VOTER last_known_addr { host: "127.6.235.125" port: 41279 } attrs { promote: false } } peers { permanent_uuid: "254f36e26f8d4ddf823cf81c7a5c2986" member_type: VOTER last_known_addr { host: "127.6.235.124" port: 33005 } attrs { promote: false } } }
I20260928 02:11:31.693152 14110 sys_catalog.cc:455] T 00000000000000000000000000000000 P 11adc61886584956adfdac31047373c1 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "11adc61886584956adfdac31047373c1" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "11adc61886584956adfdac31047373c1" member_type: VOTER last_known_addr { host: "127.6.235.126" port: 34853 } } peers { permanent_uuid: "bafe67917b9a4b40a60409c753bf4b96" member_type: VOTER last_known_addr { host: "127.6.235.125" port: 41279 } attrs { promote: false } } peers { permanent_uuid: "254f36e26f8d4ddf823cf81c7a5c2986" member_type: VOTER last_known_addr { host: "127.6.235.124" port: 33005 } attrs { promote: false } } }
I20260928 02:11:31.693228 14241 sys_catalog.cc:455] T 00000000000000000000000000000000 P 11adc61886584956adfdac31047373c1 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "11adc61886584956adfdac31047373c1" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "11adc61886584956adfdac31047373c1" member_type: VOTER last_known_addr { host: "127.6.235.126" port: 34853 } } peers { permanent_uuid: "bafe67917b9a4b40a60409c753bf4b96" member_type: VOTER last_known_addr { host: "127.6.235.125" port: 41279 } attrs { promote: false } } peers { permanent_uuid: "254f36e26f8d4ddf823cf81c7a5c2986" member_type: VOTER last_known_addr { host: "127.6.235.124" port: 33005 } attrs { promote: false } } }
I20260928 02:11:31.693529 14110 sys_catalog.cc:458] T 00000000000000000000000000000000 P 11adc61886584956adfdac31047373c1 [sys.catalog]: This master's current role is: LEADER
I20260928 02:11:31.693533 14241 sys_catalog.cc:458] T 00000000000000000000000000000000 P 11adc61886584956adfdac31047373c1 [sys.catalog]: This master's current role is: LEADER
I20260928 02:11:31.699802 14246 sys_catalog.cc:455] T 00000000000000000000000000000000 P 254f36e26f8d4ddf823cf81c7a5c2986 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "11adc61886584956adfdac31047373c1" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "11adc61886584956adfdac31047373c1" member_type: VOTER last_known_addr { host: "127.6.235.126" port: 34853 } } peers { permanent_uuid: "bafe67917b9a4b40a60409c753bf4b96" member_type: VOTER last_known_addr { host: "127.6.235.125" port: 41279 } attrs { promote: false } } peers { permanent_uuid: "254f36e26f8d4ddf823cf81c7a5c2986" member_type: VOTER last_known_addr { host: "127.6.235.124" port: 33005 } attrs { promote: false } } }
I20260928 02:11:31.701054 14246 sys_catalog.cc:458] T 00000000000000000000000000000000 P 254f36e26f8d4ddf823cf81c7a5c2986 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 02:11:31.702987 14241 sys_catalog.cc:455] T 00000000000000000000000000000000 P 11adc61886584956adfdac31047373c1 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "11adc61886584956adfdac31047373c1" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "11adc61886584956adfdac31047373c1" member_type: VOTER last_known_addr { host: "127.6.235.126" port: 34853 } } peers { permanent_uuid: "bafe67917b9a4b40a60409c753bf4b96" member_type: VOTER last_known_addr { host: "127.6.235.125" port: 41279 } attrs { promote: false } } peers { permanent_uuid: "254f36e26f8d4ddf823cf81c7a5c2986" member_type: VOTER last_known_addr { host: "127.6.235.124" port: 33005 } attrs { promote: false } } }
I20260928 02:11:31.703296 14241 sys_catalog.cc:458] T 00000000000000000000000000000000 P 11adc61886584956adfdac31047373c1 [sys.catalog]: This master's current role is: LEADER
I20260928 02:11:31.704025 13867 raft_consensus.cc:2932] T 00000000000000000000000000000000 P bafe67917b9a4b40a60409c753bf4b96 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 254f36e26f8d4ddf823cf81c7a5c2986 (127.6.235.124) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "11adc61886584956adfdac31047373c1" member_type: VOTER last_known_addr { host: "127.6.235.126" port: 34853 } } peers { permanent_uuid: "bafe67917b9a4b40a60409c753bf4b96" member_type: VOTER last_known_addr { host: "127.6.235.125" port: 41279 } attrs { promote: false } } peers { permanent_uuid: "254f36e26f8d4ddf823cf81c7a5c2986" member_type: VOTER last_known_addr { host: "127.6.235.124" port: 33005 } attrs { promote: false } } }
I20260928 02:11:31.707264 14247 sys_catalog.cc:455] T 00000000000000000000000000000000 P bafe67917b9a4b40a60409c753bf4b96 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "11adc61886584956adfdac31047373c1" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "11adc61886584956adfdac31047373c1" member_type: VOTER last_known_addr { host: "127.6.235.126" port: 34853 } } peers { permanent_uuid: "bafe67917b9a4b40a60409c753bf4b96" member_type: VOTER last_known_addr { host: "127.6.235.125" port: 41279 } attrs { promote: false } } peers { permanent_uuid: "254f36e26f8d4ddf823cf81c7a5c2986" member_type: VOTER last_known_addr { host: "127.6.235.124" port: 33005 } attrs { promote: false } } }
I20260928 02:11:31.707636 14247 sys_catalog.cc:458] T 00000000000000000000000000000000 P bafe67917b9a4b40a60409c753bf4b96 [sys.catalog]: This master's current role is: FOLLOWER
W20260928 02:11:31.786348 14239 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:11:31.786665 14239 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:11:31.786825 14239 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:11:31.795087 14239 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:11:31.795209 14239 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:11:31.795425 14239 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.235.66
I20260928 02:11:31.805940 14239 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.235.84:44519
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790561432251112-7085-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790561432251112-7085-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790561432251112-7085-0/krb5kdc/oryx_127.6.235.66.keytab
--principal=oryx/127.6.235.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.235.66:42855
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790561432251112-7085-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.6.235.66
--webserver_port=33027
--tserver_master_addrs=127.6.235.126:34853,127.6.235.125:41279,127.6.235.124:33005
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.235.66
--log_dir=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790561432251112-7085-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 00:51:59 UTC on ccycloud.quasar-axgkwn.root.comops.site
build id 4676464
ASAN enabled
I20260928 02:11:31.807405 14239 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:11:31.808965 14239 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:11:31.824338 14252 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:11:31.824249 14255 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:11:31.824342 14253 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:11:31.825706 14239 server_base.cc:1034] running on GCE node
I20260928 02:11:31.826428 14239 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:11:31.827546 14239 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:11:31.828807 14239 hybrid_clock.cc:648] HybridClock initialized: now 1790561491828750 us; error 60 us; skew 500 ppm
I20260928 02:11:31.829187 14239 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 02:11:31 dist-test-slave-86gv krb5kdc[13212](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561491, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.235.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:11:31.833105 14239 init.cc:377] Logged in from keytab as oryx/127.6.235.66@KRBTEST.COM (short username oryx)
I20260928 02:11:31.834862 14239 webserver.cc:460] Webserver started at http://127.6.235.66:33027/ using document root <none> and password file <none>
I20260928 02:11:31.835677 14239 fs_manager.cc:359] Metadata directory not provided
I20260928 02:11:31.835857 14239 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:11:31.842545 14239 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.001s
I20260928 02:11:31.846036 14262 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:11:31.847774 14239 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.001s
I20260928 02:11:31.848001 14239 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790561432251112-7085-0/minicluster-data/ts-1/data,/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790561432251112-7085-0/minicluster-data/ts-1/wal
uuid: "9f8e4ca0aeac43aa8ab6f8ab9a4d6fe9"
format_stamp: "Formatted at 2026-09-28 02:11:28 on dist-test-slave-86gv"
server_key: "b185fb7050e8c310d8982c65add9c482"
server_key_iv: "fd730cf34d10e397e130f0dc2e66479c"
server_key_version: "encryptionkey@0"
I20260928 02:11:31.849119 14239 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790561432251112-7085-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790561432251112-7085-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790561432251112-7085-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:11:31.866400 14239 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:11:31.871327 14239 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:11:31.871690 14239 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:11:31.872764 14239 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:11:31.874203 14239 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:11:31.874290 14239 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:11:31.874361 14239 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:11:31.874418 14239 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:11:31.909621 14239 rpc_server.cc:307] RPC server started. Bound to: 127.6.235.66:42855
I20260928 02:11:31.909708 14374 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.235.66:42855 every 8 connection(s)
I20260928 02:11:31.911250 14239 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790561432251112-7085-0/minicluster-data/ts-1/data/info.pb
I20260928 02:11:31.913079 7085 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTlaUuc/build/asan/bin/kudu as pid 14239
I20260928 02:11:31.913381 7085 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskTlaUuc/build/asan/bin/kudu pid 13586
Sep 28 02:11:31 dist-test-slave-86gv krb5kdc[13212](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561491, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.235.66@KRBTEST.COM for oryx/127.6.235.126@KRBTEST.COM
Sep 28 02:11:31 dist-test-slave-86gv krb5kdc[13212](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561491, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.235.66@KRBTEST.COM for oryx/127.6.235.124@KRBTEST.COM
Sep 28 02:11:31 dist-test-slave-86gv krb5kdc[13212](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561491, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.235.66@KRBTEST.COM for oryx/127.6.235.125@KRBTEST.COM
Sep 28 02:11:31 dist-test-slave-86gv krb5kdc[13212](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561486, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.6.235.67@KRBTEST.COM
I20260928 02:11:31.955331 14380 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:11:31.914785 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.235.66:33159 (local address 127.6.235.124:33005)
0928 02:11:31.921918 (+ 7133us) server_negotiation.cc:206] Beginning negotiation
0928 02:11:31.921928 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 02:11:31.929135 (+ 7207us) server_negotiation.cc:407] Connection header received
0928 02:11:31.931579 (+ 2444us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:11:31.931585 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:11:31.931700 (+ 115us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:11:31.931822 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:11:31.935096 (+ 3274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:11:31.936276 (+ 1180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:11:31.939255 (+ 2979us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:11:31.940024 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:11:31.940115 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:11:31.946456 (+ 6341us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:11:31.946465 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:11:31.946496 (+ 31us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:11:31.946548 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:11:31.951384 (+ 4836us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:11:31.952403 (+ 1019us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:11:31.952408 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:11:31.952411 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:11:31.952549 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:11:31.953283 (+ 734us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:11:31.953288 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:11:31.953291 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:11:31.953621 (+ 330us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:11:31.953758 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0928 02:11:31.954731 (+ 973us) server_negotiation.cc:299] Negotiation successful
0928 02:11:31.954950 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6908,"thread_start_us":6755,"threads_started":1}
I20260928 02:11:31.956276 14381 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:11:31.915220 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49998 (local address 127.6.235.67:40303)
0928 02:11:31.931425 (+ 16205us) server_negotiation.cc:206] Beginning negotiation
0928 02:11:31.931434 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 02:11:31.931463 (+ 29us) server_negotiation.cc:407] Connection header received
0928 02:11:31.931542 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:11:31.931552 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:11:31.931861 (+ 309us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:11:31.932046 (+ 185us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:11:31.934391 (+ 2345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:11:31.936059 (+ 1668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:11:31.937990 (+ 1931us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:11:31.939000 (+ 1010us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:11:31.939147 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:11:31.947647 (+ 8500us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:11:31.947671 (+ 24us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:11:31.947692 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:11:31.947739 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:11:31.951844 (+ 4105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:11:31.953128 (+ 1284us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:11:31.953138 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:11:31.953141 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:11:31.953319 (+ 178us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:11:31.953974 (+ 655us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:11:31.953979 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:11:31.953983 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:11:31.954326 (+ 343us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:11:31.954640 (+ 314us) server_negotiation.cc:1035] Waiting for connection context
0928 02:11:31.955600 (+ 960us) server_negotiation.cc:299] Negotiation successful
0928 02:11:31.955859 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15995,"thread_start_us":15815,"threads_started":1}
I20260928 02:11:31.956319 14384 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:11:31.915965 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.235.66:39641 (local address 127.6.235.126:34853)
0928 02:11:31.930866 (+ 14901us) server_negotiation.cc:206] Beginning negotiation
0928 02:11:31.930872 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 02:11:31.930892 (+ 20us) server_negotiation.cc:407] Connection header received
0928 02:11:31.932464 (+ 1572us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:11:31.932471 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:11:31.932556 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:11:31.932675 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:11:31.935090 (+ 2415us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:11:31.936153 (+ 1063us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:11:31.939226 (+ 3073us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:11:31.939996 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:11:31.940118 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:11:31.947004 (+ 6886us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:11:31.947012 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:11:31.947016 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:11:31.947074 (+ 58us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:11:31.950920 (+ 3846us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:11:31.952371 (+ 1451us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:11:31.952377 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:11:31.952380 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:11:31.952582 (+ 202us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:11:31.953290 (+ 708us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:11:31.953295 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:11:31.953298 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:11:31.953621 (+ 323us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:11:31.953752 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0928 02:11:31.955742 (+ 1990us) server_negotiation.cc:299] Negotiation successful
0928 02:11:31.955952 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14770,"thread_start_us":14661,"threads_started":1}
I20260928 02:11:31.956370 14383 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:11:31.915339 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.235.124:33005 (local address 127.6.235.66:33159)
0928 02:11:31.929004 (+ 13665us) negotiation.cc:107] Waiting for socket to connect
0928 02:11:31.929030 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0928 02:11:31.931205 (+ 2175us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:11:31.932471 (+ 1266us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:11:31.932484 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:11:31.933754 (+ 1270us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:11:31.934892 (+ 1138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:11:31.934912 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:11:31.936483 (+ 1571us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:11:31.936488 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:11:31.939075 (+ 2587us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:11:31.939090 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:11:31.940544 (+ 1454us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:11:31.940558 (+ 14us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:11:31.941047 (+ 489us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:11:31.942154 (+ 1107us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:11:31.942170 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:11:31.946101 (+ 3931us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:11:31.951554 (+ 5453us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:11:31.951558 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:11:31.951564 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:11:31.952253 (+ 689us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:11:31.952711 (+ 458us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:11:31.952737 (+ 26us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:11:31.952740 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:11:31.953131 (+ 391us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:11:31.953851 (+ 720us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:11:31.953859 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:11:31.954331 (+ 472us) client_negotiation.cc:769] Sending connection context
0928 02:11:31.954621 (+ 290us) client_negotiation.cc:240] Negotiation successful
0928 02:11:31.955194 (+ 573us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13462,"mutex_wait_us":575,"spinlock_wait_cycles":43904,"thread_start_us":13651,"threads_started":1}
I20260928 02:11:31.957304 14385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:11:31.930955 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.235.66:43605 (local address 127.6.235.125:41279)
0928 02:11:31.931598 (+ 643us) server_negotiation.cc:206] Beginning negotiation
0928 02:11:31.931603 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 02:11:31.932198 (+ 595us) server_negotiation.cc:407] Connection header received
0928 02:11:31.932419 (+ 221us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:11:31.932425 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:11:31.932513 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:11:31.932589 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:11:31.935095 (+ 2506us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:11:31.936276 (+ 1181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:11:31.939302 (+ 3026us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:11:31.939999 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:11:31.940110 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:11:31.946890 (+ 6780us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:11:31.946908 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:11:31.946913 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:11:31.946958 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:11:31.950834 (+ 3876us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:11:31.952134 (+ 1300us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:11:31.952143 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:11:31.952149 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:11:31.952464 (+ 315us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:11:31.953255 (+ 791us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:11:31.953260 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:11:31.953263 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:11:31.953646 (+ 383us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:11:31.953786 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0928 02:11:31.956704 (+ 2918us) server_negotiation.cc:299] Negotiation successful
0928 02:11:31.956972 (+ 268us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":531,"thread_start_us":400,"threads_started":1}
I20260928 02:11:31.957319 14386 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:11:31.931582 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.235.125:41279 (local address 127.6.235.66:43605)
0928 02:11:31.932102 (+ 520us) negotiation.cc:107] Waiting for socket to connect
0928 02:11:31.932119 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0928 02:11:31.932288 (+ 169us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:11:31.932847 (+ 559us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:11:31.932851 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:11:31.933754 (+ 903us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:11:31.934894 (+ 1140us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:11:31.934912 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:11:31.936483 (+ 1571us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:11:31.936488 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:11:31.939075 (+ 2587us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:11:31.939090 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:11:31.940579 (+ 1489us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:11:31.940586 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:11:31.941045 (+ 459us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:11:31.942100 (+ 1055us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:11:31.942132 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:11:31.946658 (+ 4526us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:11:31.951029 (+ 4371us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:11:31.951039 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:11:31.951059 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:11:31.951997 (+ 938us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:11:31.952692 (+ 695us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:11:31.952704 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:11:31.952709 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:11:31.953122 (+ 413us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:11:31.953862 (+ 740us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:11:31.953867 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:11:31.954838 (+ 971us) client_negotiation.cc:769] Sending connection context
0928 02:11:31.956570 (+ 1732us) client_negotiation.cc:240] Negotiation successful
0928 02:11:31.956871 (+ 301us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":407,"mutex_wait_us":1211,"spinlock_wait_cycles":46464,"thread_start_us":277,"threads_started":1}
I20260928 02:11:31.958134 14375 heartbeater.cc:347] Connected to a master server at 127.6.235.124:33005
I20260928 02:11:31.958303 14382 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:11:31.915312 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.235.126:34853 (local address 127.6.235.66:39641)
0928 02:11:31.927880 (+ 12568us) negotiation.cc:107] Waiting for socket to connect
0928 02:11:31.927966 (+ 86us) client_negotiation.cc:174] Beginning negotiation
0928 02:11:31.932272 (+ 4306us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:11:31.932923 (+ 651us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:11:31.932928 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:11:31.933739 (+ 811us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:11:31.934884 (+ 1145us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:11:31.934913 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:11:31.936343 (+ 1430us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:11:31.936349 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:11:31.939059 (+ 2710us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:11:31.939077 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:11:31.940545 (+ 1468us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:11:31.940559 (+ 14us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:11:31.941037 (+ 478us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:11:31.942037 (+ 1000us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:11:31.942098 (+ 61us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:11:31.945936 (+ 3838us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:11:31.951114 (+ 5178us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:11:31.951126 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:11:31.951133 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:11:31.952201 (+ 1068us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:11:31.952711 (+ 510us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:11:31.952735 (+ 24us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:11:31.952739 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:11:31.953131 (+ 392us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:11:31.953851 (+ 720us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:11:31.953859 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:11:31.955148 (+ 1289us) client_negotiation.cc:769] Sending connection context
0928 02:11:31.957564 (+ 2416us) client_negotiation.cc:240] Negotiation successful
0928 02:11:31.957845 (+ 281us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12085,"mutex_wait_us":1490,"spinlock_wait_cycles":83200,"thread_start_us":14667,"threads_started":1}
I20260928 02:11:31.958317 13709 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 02:11:31.958472 14375 heartbeater.cc:464] Registering TS with master...
I20260928 02:11:31.959574 14375 heartbeater.cc:511] Master 127.6.235.124:33005 requested a full tablet report, sending...
I20260928 02:11:31.961062 14376 heartbeater.cc:347] Connected to a master server at 127.6.235.126:34853
I20260928 02:11:31.961193 14376 heartbeater.cc:464] Registering TS with master...
I20260928 02:11:31.961591 14377 heartbeater.cc:347] Connected to a master server at 127.6.235.125:41279
I20260928 02:11:31.961678 14376 heartbeater.cc:511] Master 127.6.235.126:34853 requested a full tablet report, sending...
I20260928 02:11:31.961733 14377 heartbeater.cc:464] Registering TS with master...
I20260928 02:11:31.961975 14045 ts_manager.cc:194] Registered new tserver with Master: 9f8e4ca0aeac43aa8ab6f8ab9a4d6fe9 (127.6.235.66:42855)
I20260928 02:11:31.962311 14377 heartbeater.cc:511] Master 127.6.235.125:41279 requested a full tablet report, sending...
I20260928 02:11:31.962888 13257 ts_manager.cc:194] Re-registered known tserver with Master: 9f8e4ca0aeac43aa8ab6f8ab9a4d6fe9 (127.6.235.66:42855)
I20260928 02:11:31.963325 13857 ts_manager.cc:194] Registered new tserver with Master: 9f8e4ca0aeac43aa8ab6f8ab9a4d6fe9 (127.6.235.66:42855)
I20260928 02:11:31.964469 13257 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.6.235.66@KRBTEST.COM'} at 127.6.235.66:39641
I20260928 02:11:32.059984 7085 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTlaUuc/build/asan/bin/kudu with pid 13586
I20260928 02:11:32.077273 7085 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTlaUuc/build/asan/bin/kudu
/tmp/dist-test-taskTlaUuc/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790561432251112-7085-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790561432251112-7085-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790561432251112-7085-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790561432251112-7085-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.235.67:40303
--local_ip_for_outbound_sockets=127.6.235.67
--tserver_master_addrs=127.6.235.126:34853
--webserver_port=43293
--webserver_interface=127.6.235.67
--keytab_file=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790561432251112-7085-0/krb5kdc/oryx_127.6.235.67.keytab
--principal=oryx/127.6.235.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.235.84:44519
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.6.235.126:34853,127.6.235.125:41279
--tserver_master_addrs=127.6.235.126:34853,127.6.235.125:41279,127.6.235.124:33005 with env {KRB5CCNAME=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790561432251112-7085-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790561432251112-7085-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790561432251112-7085-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 02:11:32.318181 14228 heartbeater.cc:503] Master 127.6.235.126:34853 was elected leader, sending a full tablet report...
W20260928 02:11:32.422101 14388 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:11:32.422452 14388 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:11:32.422633 14388 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:11:32.430347 14388 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:11:32.430510 14388 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:11:32.430724 14388 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.235.67
I20260928 02:11:32.441699 14388 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.235.84:44519
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790561432251112-7085-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790561432251112-7085-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790561432251112-7085-0/krb5kdc/oryx_127.6.235.67.keytab
--principal=oryx/127.6.235.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.235.67:40303
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790561432251112-7085-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.6.235.67
--webserver_port=43293
--tserver_master_addrs=127.6.235.126:34853,127.6.235.125:41279,127.6.235.124:33005
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.235.67
--log_dir=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790561432251112-7085-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 00:51:59 UTC on ccycloud.quasar-axgkwn.root.comops.site
build id 4676464
ASAN enabled
I20260928 02:11:32.443233 14388 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:11:32.444599 14388 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:11:32.458540 14394 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:11:32.458532 14393 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:11:32.459115 14396 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:11:32.460520 14388 server_base.cc:1034] running on GCE node
I20260928 02:11:32.461290 14388 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:11:32.462399 14388 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:11:32.463644 14388 hybrid_clock.cc:648] HybridClock initialized: now 1790561492463602 us; error 42 us; skew 500 ppm
I20260928 02:11:32.463949 14388 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 02:11:32 dist-test-slave-86gv krb5kdc[13212](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561492, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.235.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:11:32.474537 14388 init.cc:377] Logged in from keytab as oryx/127.6.235.67@KRBTEST.COM (short username oryx)
I20260928 02:11:32.476267 14388 webserver.cc:460] Webserver started at http://127.6.235.67:43293/ using document root <none> and password file <none>
I20260928 02:11:32.477283 14388 fs_manager.cc:359] Metadata directory not provided
I20260928 02:11:32.477453 14388 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:11:32.483242 14388 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20260928 02:11:32.486271 14403 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:11:32.488215 14388 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260928 02:11:32.488392 14388 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790561432251112-7085-0/minicluster-data/ts-2/data,/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790561432251112-7085-0/minicluster-data/ts-2/wal
uuid: "9dae072a16c44f1a8e4fe720ecff93cc"
format_stamp: "Formatted at 2026-09-28 02:11:29 on dist-test-slave-86gv"
server_key: "4e53bee3367eb30daf885b3a1365e5f8"
server_key_iv: "0d61598ee067e647ab16c18947542468"
server_key_version: "encryptionkey@0"
I20260928 02:11:32.489842 14388 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790561432251112-7085-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790561432251112-7085-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790561432251112-7085-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:11:32.514176 14388 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:11:32.518872 14388 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:11:32.519279 14388 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:11:32.520336 14388 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:11:32.521780 14388 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:11:32.521870 14388 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:11:32.521994 14388 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:11:32.522075 14388 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:11:32.556651 14388 rpc_server.cc:307] RPC server started. Bound to: 127.6.235.67:40303
I20260928 02:11:32.556770 14515 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.235.67:40303 every 8 connection(s)
I20260928 02:11:32.558171 14388 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790561432251112-7085-0/minicluster-data/ts-2/data/info.pb
I20260928 02:11:32.560158 7085 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTlaUuc/build/asan/bin/kudu as pid 14388
Sep 28 02:11:32 dist-test-slave-86gv krb5kdc[13212](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561492, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.235.67@KRBTEST.COM for oryx/127.6.235.125@KRBTEST.COM
Sep 28 02:11:32 dist-test-slave-86gv krb5kdc[13212](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561492, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.235.67@KRBTEST.COM for oryx/127.6.235.124@KRBTEST.COM
I20260928 02:11:32.624493 14521 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:11:32.561648 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.235.67:51603 (local address 127.6.235.125:41279)
0928 02:11:32.588298 (+ 26650us) server_negotiation.cc:206] Beginning negotiation
0928 02:11:32.588320 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0928 02:11:32.590541 (+ 2221us) server_negotiation.cc:407] Connection header received
0928 02:11:32.597448 (+ 6907us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:11:32.597458 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:11:32.597581 (+ 123us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:11:32.597683 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:11:32.604412 (+ 6729us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:11:32.605762 (+ 1350us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:11:32.608546 (+ 2784us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:11:32.609229 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:11:32.609334 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:11:32.616416 (+ 7082us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:11:32.616429 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:11:32.616433 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:11:32.616494 (+ 61us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:11:32.620335 (+ 3841us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:11:32.621482 (+ 1147us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:11:32.621491 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:11:32.621497 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:11:32.621659 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:11:32.622268 (+ 609us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:11:32.622273 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:11:32.622277 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:11:32.622663 (+ 386us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:11:32.622804 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0928 02:11:32.623887 (+ 1083us) server_negotiation.cc:299] Negotiation successful
0928 02:11:32.624091 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26494,"thread_start_us":26672,"threads_started":1}
I20260928 02:11:32.625222 14532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:11:32.587745 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41032 (local address 127.6.235.124:33005)
0928 02:11:32.604020 (+ 16275us) server_negotiation.cc:206] Beginning negotiation
0928 02:11:32.604027 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 02:11:32.604049 (+ 22us) server_negotiation.cc:407] Connection header received
0928 02:11:32.604110 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:11:32.604114 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:11:32.604213 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:11:32.606351 (+ 2138us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:11:32.606537 (+ 186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:11:32.607635 (+ 1098us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:11:32.611053 (+ 3418us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:11:32.611730 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:11:32.612625 (+ 895us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:11:32.616575 (+ 3950us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:11:32.616582 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:11:32.616589 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:11:32.616622 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:11:32.621441 (+ 4819us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:11:32.622260 (+ 819us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:11:32.622265 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:11:32.622269 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:11:32.622405 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:11:32.622919 (+ 514us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:11:32.622924 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:11:32.622927 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:11:32.623249 (+ 322us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:11:32.624497 (+ 1248us) server_negotiation.cc:1035] Waiting for connection context
0928 02:11:32.624649 (+ 152us) server_negotiation.cc:299] Negotiation successful
0928 02:11:32.624875 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16132,"thread_start_us":16567,"threads_started":1}
I20260928 02:11:32.625416 14520 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:11:32.571640 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42958 (local address 127.6.235.126:34853)
0928 02:11:32.587349 (+ 15709us) server_negotiation.cc:206] Beginning negotiation
0928 02:11:32.587358 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 02:11:32.587387 (+ 29us) server_negotiation.cc:407] Connection header received
0928 02:11:32.587468 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:11:32.587476 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:11:32.587577 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:11:32.587720 (+ 143us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:11:32.592876 (+ 5156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:11:32.594497 (+ 1621us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:11:32.597660 (+ 3163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:11:32.598517 (+ 857us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:11:32.598672 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:11:32.606872 (+ 8200us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:11:32.606883 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:11:32.606891 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:11:32.606952 (+ 61us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:11:32.619401 (+ 12449us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:11:32.622304 (+ 2903us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:11:32.622311 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:11:32.622315 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:11:32.622494 (+ 179us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:11:32.623041 (+ 547us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:11:32.623049 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:11:32.623053 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:11:32.623434 (+ 381us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:11:32.624657 (+ 1223us) server_negotiation.cc:1035] Waiting for connection context
0928 02:11:32.624776 (+ 119us) server_negotiation.cc:299] Negotiation successful
0928 02:11:32.625047 (+ 271us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15527,"thread_start_us":15478,"threads_started":1}
I20260928 02:11:32.626842 14524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:11:32.562638 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.235.125:41279 (local address 127.6.235.67:51603)
0928 02:11:32.589847 (+ 27209us) negotiation.cc:107] Waiting for socket to connect
0928 02:11:32.589905 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0928 02:11:32.597034 (+ 7129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:11:32.598252 (+ 1218us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:11:32.598271 (+ 19us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:11:32.603609 (+ 5338us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:11:32.604138 (+ 529us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:11:32.604186 (+ 48us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:11:32.606059 (+ 1873us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:11:32.606068 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:11:32.608364 (+ 2296us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:11:32.608384 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:11:32.609715 (+ 1331us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:11:32.609724 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:11:32.610062 (+ 338us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:11:32.610780 (+ 718us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:11:32.610811 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:11:32.616121 (+ 5310us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:11:32.620514 (+ 4393us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:11:32.620525 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:11:32.620547 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:11:32.621325 (+ 778us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:11:32.621830 (+ 505us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:11:32.621835 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:11:32.621839 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:11:32.622108 (+ 269us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:11:32.622919 (+ 811us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:11:32.622928 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:11:32.623435 (+ 507us) client_negotiation.cc:769] Sending connection context
0928 02:11:32.624703 (+ 1268us) client_negotiation.cc:240] Negotiation successful
0928 02:11:32.625331 (+ 628us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":26965,"thread_start_us":27317,"threads_started":1}
Sep 28 02:11:32 dist-test-slave-86gv krb5kdc[13212](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561492, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.235.67@KRBTEST.COM for oryx/127.6.235.126@KRBTEST.COM
I20260928 02:11:32.628784 14518 heartbeater.cc:347] Connected to a master server at 127.6.235.125:41279
I20260928 02:11:32.629173 14518 heartbeater.cc:464] Registering TS with master...
I20260928 02:11:32.630156 14518 heartbeater.cc:511] Master 127.6.235.125:41279 requested a full tablet report, sending...
I20260928 02:11:32.632455 13857 ts_manager.cc:194] Registered new tserver with Master: 9dae072a16c44f1a8e4fe720ecff93cc (127.6.235.67:40303)
I20260928 02:11:32.633759 14533 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:11:32.588516 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60162 (local address 127.6.235.125:41279)
0928 02:11:32.601571 (+ 13055us) server_negotiation.cc:206] Beginning negotiation
0928 02:11:32.601580 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 02:11:32.601605 (+ 25us) server_negotiation.cc:407] Connection header received
0928 02:11:32.601688 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:11:32.601695 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:11:32.601790 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:11:32.604764 (+ 2974us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:11:32.605125 (+ 361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:11:32.606543 (+ 1418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:11:32.615930 (+ 9387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:11:32.616670 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:11:32.616811 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:11:32.620833 (+ 4022us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:11:32.620840 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:11:32.620845 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:11:32.620881 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:11:32.629578 (+ 8697us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:11:32.630515 (+ 937us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:11:32.630520 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:11:32.630523 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:11:32.630675 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:11:32.631126 (+ 451us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:11:32.631130 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:11:32.631133 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:11:32.631493 (+ 360us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:11:32.632879 (+ 1386us) server_negotiation.cc:1035] Waiting for connection context
0928 02:11:32.633069 (+ 190us) server_negotiation.cc:299] Negotiation successful
0928 02:11:32.633338 (+ 269us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12865,"thread_start_us":13427,"threads_started":1}
I20260928 02:11:32.636641 14534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:11:32.590630 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.235.67:49689 (local address 127.6.235.124:33005)
0928 02:11:32.608415 (+ 17785us) server_negotiation.cc:206] Beginning negotiation
0928 02:11:32.608421 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 02:11:32.608439 (+ 18us) server_negotiation.cc:407] Connection header received
0928 02:11:32.608508 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:11:32.608513 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:11:32.608598 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:11:32.608700 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:11:32.610436 (+ 1736us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:11:32.612135 (+ 1699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:11:32.613996 (+ 1861us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:11:32.614853 (+ 857us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:11:32.614986 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:11:32.618280 (+ 3294us) spinlock_profiling.cc:243] Waited 2.32 ms on lock 0x60c0000347b8. stack: 00007fd1fbbb766e 00007fd1fbbb7430 00007fd1fac3e5ed 00000000008ea064 00000000008ea02d 00000000008e99fa 00007fd1fbbad0fd 00007fd1fbbacd1e 00007fd2026c4664 00007fd2026dee8d 00007fd2026d9f76 00007fd2026d38a5 00007fd202641ea5 00007fd202640eff 00007fd20267482f 00007fd20267458d
0928 02:11:32.624917 (+ 6637us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:11:32.624927 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:11:32.624931 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:11:32.624974 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:11:32.632491 (+ 7517us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:11:32.633569 (+ 1078us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:11:32.633576 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:11:32.633581 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:11:32.633745 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:11:32.634433 (+ 688us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:11:32.634438 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:11:32.634442 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:11:32.634806 (+ 364us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:11:32.634944 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0928 02:11:32.635996 (+ 1052us) server_negotiation.cc:299] Negotiation successful
0928 02:11:32.636235 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17648,"spinlock_wait_cycles":5328640,"thread_start_us":19622,"threads_started":1}
I20260928 02:11:32.637214 14535 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:11:32.596520 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.235.124:33005 (local address 127.6.235.67:49689)
0928 02:11:32.605853 (+ 9333us) negotiation.cc:107] Waiting for socket to connect
0928 02:11:32.605880 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0928 02:11:32.606004 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:11:32.608948 (+ 2944us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:11:32.608954 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:11:32.609756 (+ 802us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:11:32.610148 (+ 392us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:11:32.610162 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:11:32.612290 (+ 2128us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:11:32.612295 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:11:32.613838 (+ 1543us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:11:32.613853 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:11:32.615080 (+ 1227us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:11:32.615089 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:11:32.615298 (+ 209us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:11:32.615934 (+ 636us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:11:32.615960 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:11:32.624339 (+ 8379us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:11:32.632729 (+ 8390us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:11:32.632735 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:11:32.632740 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:11:32.633411 (+ 671us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:11:32.634133 (+ 722us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:11:32.634139 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:11:32.634143 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:11:32.634302 (+ 159us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:11:32.635005 (+ 703us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:11:32.635010 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:11:32.635170 (+ 160us) client_negotiation.cc:769] Sending connection context
0928 02:11:32.635307 (+ 137us) client_negotiation.cc:240] Negotiation successful
0928 02:11:32.635614 (+ 307us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9186,"thread_start_us":9558,"threads_started":1}
I20260928 02:11:32.638116 14516 heartbeater.cc:347] Connected to a master server at 127.6.235.124:33005
I20260928 02:11:32.638281 14516 heartbeater.cc:464] Registering TS with master...
I20260928 02:11:32.638674 14516 heartbeater.cc:511] Master 127.6.235.124:33005 requested a full tablet report, sending...
I20260928 02:11:32.630223 13257 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42958:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260928 02:11:32.639972 14045 ts_manager.cc:194] Registered new tserver with Master: 9dae072a16c44f1a8e4fe720ecff93cc (127.6.235.67:40303)
I20260928 02:11:32.640594 14528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:11:32.561364 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.235.67:51611 (local address 127.6.235.126:34853)
0928 02:11:32.587111 (+ 25747us) server_negotiation.cc:206] Beginning negotiation
0928 02:11:32.587117 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 02:11:32.597286 (+ 10169us) server_negotiation.cc:407] Connection header received
0928 02:11:32.597625 (+ 339us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:11:32.597632 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:11:32.597722 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:11:32.597821 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:11:32.609582 (+ 11761us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:11:32.610787 (+ 1205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:11:32.615437 (+ 4650us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:11:32.616086 (+ 649us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:11:32.616216 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:11:32.631050 (+ 14834us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:11:32.631058 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:11:32.631068 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:11:32.631112 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:11:32.635817 (+ 4705us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:11:32.637648 (+ 1831us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:11:32.637656 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:11:32.637658 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:11:32.637792 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:11:32.638913 (+ 1121us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:11:32.638918 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:11:32.638921 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:11:32.639315 (+ 394us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:11:32.639571 (+ 256us) server_negotiation.cc:1035] Waiting for connection context
0928 02:11:32.639983 (+ 412us) server_negotiation.cc:299] Negotiation successful
0928 02:11:32.640190 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25600,"thread_start_us":27280,"threads_started":1}
I20260928 02:11:32.640836 14523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:11:32.562004 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.235.126:34853 (local address 127.6.235.67:51611)
0928 02:11:32.590622 (+ 28618us) negotiation.cc:107] Waiting for socket to connect
0928 02:11:32.590640 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0928 02:11:32.594880 (+ 4240us) spinlock_profiling.cc:243] Waited 970 us on lock 0x61200002c628. stack: 00007f6d7e2c266e 00007f6d7e2c2430 00007f6d7d3495ed 00000000008ea064 00000000008ea02d 00000000008e99fa 00007f6d7e2b80fd 00007f6d7e2b7d1e 00007f6d911c53cd 00007f6d911c536c 00007f6d91191b3b 00007f6d84cc7536 00007f6d84cc5d9e 00007f6d84d4ddca 00007f6d84d4bf68 00007f6d84d7f82f
0928 02:11:32.597502 (+ 2622us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:11:32.598923 (+ 1421us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:11:32.598930 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:11:32.600051 (+ 1121us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:11:32.608936 (+ 8885us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:11:32.608947 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:11:32.611143 (+ 2196us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:11:32.611151 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:11:32.615030 (+ 3879us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:11:32.615047 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:11:32.619618 (+ 4571us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:11:32.619627 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:11:32.619830 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:11:32.620873 (+ 1043us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:11:32.620929 (+ 56us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:11:32.630800 (+ 9871us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:11:32.636020 (+ 5220us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:11:32.636031 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:11:32.636034 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:11:32.637487 (+ 1453us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:11:32.637924 (+ 437us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:11:32.637929 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:11:32.637932 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:11:32.638115 (+ 183us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:11:32.639542 (+ 1427us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:11:32.639547 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:11:32.639707 (+ 160us) client_negotiation.cc:769] Sending connection context
0928 02:11:32.639858 (+ 151us) client_negotiation.cc:240] Negotiation successful
0928 02:11:32.640321 (+ 463us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":27037,"spinlock_wait_cycles":2230656,"thread_start_us":26771,"threads_started":1}
W20260928 02:11:32.641466 13257 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 02:11:32.641685 14517 heartbeater.cc:347] Connected to a master server at 127.6.235.126:34853
I20260928 02:11:32.641829 14517 heartbeater.cc:464] Registering TS with master...
I20260928 02:11:32.642314 14517 heartbeater.cc:511] Master 127.6.235.126:34853 requested a full tablet report, sending...
I20260928 02:11:32.643376 13256 ts_manager.cc:194] Re-registered known tserver with Master: 9dae072a16c44f1a8e4fe720ecff93cc (127.6.235.67:40303)
I20260928 02:11:32.645085 13256 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.6.235.67@KRBTEST.COM'} at 127.6.235.67:51611
I20260928 02:11:32.682286 14549 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:11:32.670194 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36176 (local address 127.6.235.65:42113)
0928 02:11:32.670795 (+ 601us) server_negotiation.cc:206] Beginning negotiation
0928 02:11:32.670808 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0928 02:11:32.673752 (+ 2944us) server_negotiation.cc:407] Connection header received
0928 02:11:32.673946 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:11:32.673956 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:11:32.674253 (+ 297us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:11:32.674469 (+ 216us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:11:32.675398 (+ 929us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:11:32.676940 (+ 1542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:11:32.679796 (+ 2856us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:11:32.681079 (+ 1283us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:11:32.681246 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:11:32.681490 (+ 244us) server_negotiation.cc:1035] Waiting for connection context
0928 02:11:32.681785 (+ 295us) server_negotiation.cc:299] Negotiation successful
0928 02:11:32.681950 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":380,"thread_start_us":369,"threads_started":1}
I20260928 02:11:32.682390 14546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:11:32.669805 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.235.65:42113 (local address 127.0.0.1:36176)
0928 02:11:32.673612 (+ 3807us) negotiation.cc:107] Waiting for socket to connect
0928 02:11:32.673627 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 02:11:32.673795 (+ 168us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:11:32.674737 (+ 942us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:11:32.674741 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:11:32.674761 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:11:32.675114 (+ 353us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:11:32.675127 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:11:32.677122 (+ 1995us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:11:32.677127 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:11:32.679612 (+ 2485us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:11:32.679627 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:11:32.681317 (+ 1690us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:11:32.681324 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:11:32.681519 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:11:32.681610 (+ 91us) client_negotiation.cc:769] Sending connection context
0928 02:11:32.681733 (+ 123us) client_negotiation.cc:240] Negotiation successful
0928 02:11:32.681882 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3688,"mutex_wait_us":74,"thread_start_us":3343,"threads_started":1}
I20260928 02:11:32.682562 14548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:11:32.669461 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.235.66:42855 (local address 127.0.0.1:50108)
0928 02:11:32.673571 (+ 4110us) negotiation.cc:107] Waiting for socket to connect
0928 02:11:32.673602 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0928 02:11:32.673875 (+ 273us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:11:32.675138 (+ 1263us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:11:32.675145 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:11:32.675228 (+ 83us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:11:32.675782 (+ 554us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:11:32.675799 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:11:32.677647 (+ 1848us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:11:32.677652 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:11:32.679587 (+ 1935us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:11:32.679602 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:11:32.681639 (+ 2037us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:11:32.681647 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:11:32.681825 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:11:32.681859 (+ 34us) client_negotiation.cc:769] Sending connection context
0928 02:11:32.681965 (+ 106us) client_negotiation.cc:240] Negotiation successful
0928 02:11:32.682111 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3910,"mutex_wait_us":1,"thread_start_us":3877,"threads_started":1}
I20260928 02:11:32.682956 14547 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:11:32.669701 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50108 (local address 127.6.235.66:42855)
0928 02:11:32.670178 (+ 477us) server_negotiation.cc:206] Beginning negotiation
0928 02:11:32.670190 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0928 02:11:32.673742 (+ 3552us) server_negotiation.cc:407] Connection header received
0928 02:11:32.674043 (+ 301us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:11:32.674050 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:11:32.674252 (+ 202us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:11:32.674472 (+ 220us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:11:32.675984 (+ 1512us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:11:32.677478 (+ 1494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:11:32.679738 (+ 2260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:11:32.681082 (+ 1344us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:11:32.681252 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:11:32.681472 (+ 220us) server_negotiation.cc:1035] Waiting for connection context
0928 02:11:32.682579 (+ 1107us) server_negotiation.cc:299] Negotiation successful
0928 02:11:32.682768 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":294,"thread_start_us":671,"threads_started":1}
I20260928 02:11:32.687657 14162 tablet_service.cc:1467] Processing CreateTablet for tablet 32ea9599979a4296accb4041d4587d45 (DEFAULT_TABLE table=test-table [id=975490bb04b249b69d5af592326250be]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:11:32.687645 14310 tablet_service.cc:1467] Processing CreateTablet for tablet 32ea9599979a4296accb4041d4587d45 (DEFAULT_TABLE table=test-table [id=975490bb04b249b69d5af592326250be]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:11:32.689040 14162 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 32ea9599979a4296accb4041d4587d45. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:11:32.689033 14310 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 32ea9599979a4296accb4041d4587d45. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:11:32.690047 14544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:11:32.668663 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.235.67:40303 (local address 127.0.0.1:50010)
0928 02:11:32.669382 (+ 719us) negotiation.cc:107] Waiting for socket to connect
0928 02:11:32.669402 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0928 02:11:32.669519 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:11:32.677076 (+ 7557us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:11:32.677081 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:11:32.677106 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:11:32.677635 (+ 529us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:11:32.677646 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:11:32.681070 (+ 3424us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:11:32.681075 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:11:32.683154 (+ 2079us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:11:32.683167 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:11:32.689139 (+ 5972us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:11:32.689146 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:11:32.689372 (+ 226us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:11:32.689415 (+ 43us) client_negotiation.cc:769] Sending connection context
0928 02:11:32.689532 (+ 117us) client_negotiation.cc:240] Negotiation successful
0928 02:11:32.689703 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":567,"thread_start_us":408,"threads_started":1}
I20260928 02:11:32.690199 14545 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:11:32.669054 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50010 (local address 127.6.235.67:40303)
0928 02:11:32.676464 (+ 7410us) server_negotiation.cc:206] Beginning negotiation
0928 02:11:32.676471 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 02:11:32.676499 (+ 28us) server_negotiation.cc:407] Connection header received
0928 02:11:32.676580 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:11:32.676586 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:11:32.676789 (+ 203us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:11:32.677112 (+ 323us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:11:32.677909 (+ 797us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:11:32.679582 (+ 1673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:11:32.683326 (+ 3744us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:11:32.684924 (+ 1598us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:11:32.685064 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:11:32.685296 (+ 232us) server_negotiation.cc:1035] Waiting for connection context
0928 02:11:32.689665 (+ 4369us) server_negotiation.cc:299] Negotiation successful
0928 02:11:32.689865 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7240,"thread_start_us":7062,"threads_started":1}
I20260928 02:11:32.694620 14451 tablet_service.cc:1467] Processing CreateTablet for tablet 32ea9599979a4296accb4041d4587d45 (DEFAULT_TABLE table=test-table [id=975490bb04b249b69d5af592326250be]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:11:32.696199 14451 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 32ea9599979a4296accb4041d4587d45. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:11:32.702569 14550 tablet_bootstrap.cc:492] T 32ea9599979a4296accb4041d4587d45 P 9f8e4ca0aeac43aa8ab6f8ab9a4d6fe9: Bootstrap starting.
I20260928 02:11:32.705084 14551 tablet_bootstrap.cc:492] T 32ea9599979a4296accb4041d4587d45 P db7477740b844e9c8e8d14f8724f78d3: Bootstrap starting.
I20260928 02:11:32.706199 14550 tablet_bootstrap.cc:654] T 32ea9599979a4296accb4041d4587d45 P 9f8e4ca0aeac43aa8ab6f8ab9a4d6fe9: Neither blocks nor log segments found. Creating new log.
I20260928 02:11:32.707558 14550 log.cc:824] T 32ea9599979a4296accb4041d4587d45 P 9f8e4ca0aeac43aa8ab6f8ab9a4d6fe9: Log is configured to *not* fsync() on all Append() calls
I20260928 02:11:32.708705 14551 tablet_bootstrap.cc:654] T 32ea9599979a4296accb4041d4587d45 P db7477740b844e9c8e8d14f8724f78d3: Neither blocks nor log segments found. Creating new log.
I20260928 02:11:32.710116 14551 log.cc:824] T 32ea9599979a4296accb4041d4587d45 P db7477740b844e9c8e8d14f8724f78d3: Log is configured to *not* fsync() on all Append() calls
I20260928 02:11:32.711050 14550 tablet_bootstrap.cc:492] T 32ea9599979a4296accb4041d4587d45 P 9f8e4ca0aeac43aa8ab6f8ab9a4d6fe9: No bootstrap required, opened a new log
I20260928 02:11:32.711412 14550 ts_tablet_manager.cc:1397] T 32ea9599979a4296accb4041d4587d45 P 9f8e4ca0aeac43aa8ab6f8ab9a4d6fe9: Time spent bootstrapping tablet: real 0.010s user 0.007s sys 0.000s
I20260928 02:11:32.711385 14553 tablet_bootstrap.cc:492] T 32ea9599979a4296accb4041d4587d45 P 9dae072a16c44f1a8e4fe720ecff93cc: Bootstrap starting.
I20260928 02:11:32.715438 14553 tablet_bootstrap.cc:654] T 32ea9599979a4296accb4041d4587d45 P 9dae072a16c44f1a8e4fe720ecff93cc: Neither blocks nor log segments found. Creating new log.
I20260928 02:11:32.716312 14551 tablet_bootstrap.cc:492] T 32ea9599979a4296accb4041d4587d45 P db7477740b844e9c8e8d14f8724f78d3: No bootstrap required, opened a new log
I20260928 02:11:32.716562 14553 log.cc:824] T 32ea9599979a4296accb4041d4587d45 P 9dae072a16c44f1a8e4fe720ecff93cc: Log is configured to *not* fsync() on all Append() calls
I20260928 02:11:32.716609 14551 ts_tablet_manager.cc:1397] T 32ea9599979a4296accb4041d4587d45 P db7477740b844e9c8e8d14f8724f78d3: Time spent bootstrapping tablet: real 0.012s user 0.006s sys 0.003s
I20260928 02:11:32.723052 14550 raft_consensus.cc:348] T 32ea9599979a4296accb4041d4587d45 P 9f8e4ca0aeac43aa8ab6f8ab9a4d6fe9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dae072a16c44f1a8e4fe720ecff93cc" member_type: VOTER last_known_addr { host: "127.6.235.67" port: 40303 } } peers { permanent_uuid: "9f8e4ca0aeac43aa8ab6f8ab9a4d6fe9" member_type: VOTER last_known_addr { host: "127.6.235.66" port: 42855 } } peers { permanent_uuid: "db7477740b844e9c8e8d14f8724f78d3" member_type: VOTER last_known_addr { host: "127.6.235.65" port: 42113 } }
I20260928 02:11:32.723563 14550 raft_consensus.cc:374] T 32ea9599979a4296accb4041d4587d45 P 9f8e4ca0aeac43aa8ab6f8ab9a4d6fe9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:11:32.723663 14550 raft_consensus.cc:729] T 32ea9599979a4296accb4041d4587d45 P 9f8e4ca0aeac43aa8ab6f8ab9a4d6fe9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9f8e4ca0aeac43aa8ab6f8ab9a4d6fe9, State: Initialized, Role: FOLLOWER
I20260928 02:11:32.723729 14553 tablet_bootstrap.cc:492] T 32ea9599979a4296accb4041d4587d45 P 9dae072a16c44f1a8e4fe720ecff93cc: No bootstrap required, opened a new log
I20260928 02:11:32.724073 14553 ts_tablet_manager.cc:1397] T 32ea9599979a4296accb4041d4587d45 P 9dae072a16c44f1a8e4fe720ecff93cc: Time spent bootstrapping tablet: real 0.013s user 0.006s sys 0.003s
I20260928 02:11:32.724380 14550 consensus_queue.cc:260] T 32ea9599979a4296accb4041d4587d45 P 9f8e4ca0aeac43aa8ab6f8ab9a4d6fe9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dae072a16c44f1a8e4fe720ecff93cc" member_type: VOTER last_known_addr { host: "127.6.235.67" port: 40303 } } peers { permanent_uuid: "9f8e4ca0aeac43aa8ab6f8ab9a4d6fe9" member_type: VOTER last_known_addr { host: "127.6.235.66" port: 42855 } } peers { permanent_uuid: "db7477740b844e9c8e8d14f8724f78d3" member_type: VOTER last_known_addr { host: "127.6.235.65" port: 42113 } }
I20260928 02:11:32.725657 14376 heartbeater.cc:503] Master 127.6.235.126:34853 was elected leader, sending a full tablet report...
I20260928 02:11:32.726409 14550 ts_tablet_manager.cc:1428] T 32ea9599979a4296accb4041d4587d45 P 9f8e4ca0aeac43aa8ab6f8ab9a4d6fe9: Time spent starting tablet: real 0.015s user 0.010s sys 0.004s
I20260928 02:11:32.728611 14551 raft_consensus.cc:348] T 32ea9599979a4296accb4041d4587d45 P db7477740b844e9c8e8d14f8724f78d3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dae072a16c44f1a8e4fe720ecff93cc" member_type: VOTER last_known_addr { host: "127.6.235.67" port: 40303 } } peers { permanent_uuid: "9f8e4ca0aeac43aa8ab6f8ab9a4d6fe9" member_type: VOTER last_known_addr { host: "127.6.235.66" port: 42855 } } peers { permanent_uuid: "db7477740b844e9c8e8d14f8724f78d3" member_type: VOTER last_known_addr { host: "127.6.235.65" port: 42113 } }
I20260928 02:11:32.729168 14551 raft_consensus.cc:374] T 32ea9599979a4296accb4041d4587d45 P db7477740b844e9c8e8d14f8724f78d3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:11:32.729285 14551 raft_consensus.cc:729] T 32ea9599979a4296accb4041d4587d45 P db7477740b844e9c8e8d14f8724f78d3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db7477740b844e9c8e8d14f8724f78d3, State: Initialized, Role: FOLLOWER
I20260928 02:11:32.729966 14551 consensus_queue.cc:260] T 32ea9599979a4296accb4041d4587d45 P db7477740b844e9c8e8d14f8724f78d3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dae072a16c44f1a8e4fe720ecff93cc" member_type: VOTER last_known_addr { host: "127.6.235.67" port: 40303 } } peers { permanent_uuid: "9f8e4ca0aeac43aa8ab6f8ab9a4d6fe9" member_type: VOTER last_known_addr { host: "127.6.235.66" port: 42855 } } peers { permanent_uuid: "db7477740b844e9c8e8d14f8724f78d3" member_type: VOTER last_known_addr { host: "127.6.235.65" port: 42113 } }
I20260928 02:11:32.735706 14551 ts_tablet_manager.cc:1428] T 32ea9599979a4296accb4041d4587d45 P db7477740b844e9c8e8d14f8724f78d3: Time spent starting tablet: real 0.019s user 0.015s sys 0.004s
I20260928 02:11:32.736501 14553 raft_consensus.cc:348] T 32ea9599979a4296accb4041d4587d45 P 9dae072a16c44f1a8e4fe720ecff93cc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dae072a16c44f1a8e4fe720ecff93cc" member_type: VOTER last_known_addr { host: "127.6.235.67" port: 40303 } } peers { permanent_uuid: "9f8e4ca0aeac43aa8ab6f8ab9a4d6fe9" member_type: VOTER last_known_addr { host: "127.6.235.66" port: 42855 } } peers { permanent_uuid: "db7477740b844e9c8e8d14f8724f78d3" member_type: VOTER last_known_addr { host: "127.6.235.65" port: 42113 } }
I20260928 02:11:32.737000 14553 raft_consensus.cc:374] T 32ea9599979a4296accb4041d4587d45 P 9dae072a16c44f1a8e4fe720ecff93cc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:11:32.737106 14553 raft_consensus.cc:729] T 32ea9599979a4296accb4041d4587d45 P 9dae072a16c44f1a8e4fe720ecff93cc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9dae072a16c44f1a8e4fe720ecff93cc, State: Initialized, Role: FOLLOWER
I20260928 02:11:32.737782 14553 consensus_queue.cc:260] T 32ea9599979a4296accb4041d4587d45 P 9dae072a16c44f1a8e4fe720ecff93cc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dae072a16c44f1a8e4fe720ecff93cc" member_type: VOTER last_known_addr { host: "127.6.235.67" port: 40303 } } peers { permanent_uuid: "9f8e4ca0aeac43aa8ab6f8ab9a4d6fe9" member_type: VOTER last_known_addr { host: "127.6.235.66" port: 42855 } } peers { permanent_uuid: "db7477740b844e9c8e8d14f8724f78d3" member_type: VOTER last_known_addr { host: "127.6.235.65" port: 42113 } }
I20260928 02:11:32.740260 14553 ts_tablet_manager.cc:1428] T 32ea9599979a4296accb4041d4587d45 P 9dae072a16c44f1a8e4fe720ecff93cc: Time spent starting tablet: real 0.016s user 0.015s sys 0.000s
I20260928 02:11:32.740999 14517 heartbeater.cc:503] Master 127.6.235.126:34853 was elected leader, sending a full tablet report...
W20260928 02:11:32.804258 14235 tablet.cc:2369] T 32ea9599979a4296accb4041d4587d45 P db7477740b844e9c8e8d14f8724f78d3: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 02:11:32.810161 14557 raft_consensus.cc:482] T 32ea9599979a4296accb4041d4587d45 P db7477740b844e9c8e8d14f8724f78d3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 02:11:32.810484 14557 raft_consensus.cc:504] T 32ea9599979a4296accb4041d4587d45 P db7477740b844e9c8e8d14f8724f78d3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dae072a16c44f1a8e4fe720ecff93cc" member_type: VOTER last_known_addr { host: "127.6.235.67" port: 40303 } } peers { permanent_uuid: "9f8e4ca0aeac43aa8ab6f8ab9a4d6fe9" member_type: VOTER last_known_addr { host: "127.6.235.66" port: 42855 } } peers { permanent_uuid: "db7477740b844e9c8e8d14f8724f78d3" member_type: VOTER last_known_addr { host: "127.6.235.65" port: 42113 } }
I20260928 02:11:32.812373 14557 leader_election.cc:288] T 32ea9599979a4296accb4041d4587d45 P db7477740b844e9c8e8d14f8724f78d3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9dae072a16c44f1a8e4fe720ecff93cc (127.6.235.67:40303), 9f8e4ca0aeac43aa8ab6f8ab9a4d6fe9 (127.6.235.66:42855)
W20260928 02:11:32.816674 14519 tablet.cc:2369] T 32ea9599979a4296accb4041d4587d45 P 9dae072a16c44f1a8e4fe720ecff93cc: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 02:11:32.818953 14545 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:11:32.812746 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.235.65:48257 (local address 127.6.235.67:40303)
0928 02:11:32.813175 (+ 429us) server_negotiation.cc:206] Beginning negotiation
0928 02:11:32.813185 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 02:11:32.813442 (+ 257us) server_negotiation.cc:407] Connection header received
0928 02:11:32.813720 (+ 278us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:11:32.813728 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:11:32.813841 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:11:32.813937 (+ 96us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:11:32.814609 (+ 672us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:11:32.815320 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:11:32.817228 (+ 1908us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:11:32.818015 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:11:32.818216 (+ 201us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:11:32.818293 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0928 02:11:32.818555 (+ 262us) server_negotiation.cc:299] Negotiation successful
0928 02:11:32.818695 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":90}
I20260928 02:11:32.819051 14559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:11:32.812582 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.235.67:40303 (local address 127.6.235.65:48257)
0928 02:11:32.813328 (+ 746us) negotiation.cc:107] Waiting for socket to connect
0928 02:11:32.813345 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0928 02:11:32.813530 (+ 185us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:11:32.814071 (+ 541us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:11:32.814075 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:11:32.814140 (+ 65us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:11:32.814449 (+ 309us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:11:32.814458 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:11:32.815464 (+ 1006us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:11:32.815472 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:11:32.817074 (+ 1602us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:11:32.817086 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:11:32.818226 (+ 1140us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:11:32.818232 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:11:32.818391 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:11:32.818414 (+ 23us) client_negotiation.cc:769] Sending connection context
0928 02:11:32.818520 (+ 106us) client_negotiation.cc:240] Negotiation successful
0928 02:11:32.818693 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":615,"thread_start_us":418,"threads_started":1}
I20260928 02:11:32.819700 14560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:11:32.813414 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.235.66:42855 (local address 127.6.235.65:58391)
0928 02:11:32.814080 (+ 666us) negotiation.cc:107] Waiting for socket to connect
0928 02:11:32.814098 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0928 02:11:32.814215 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:11:32.814562 (+ 347us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:11:32.814566 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:11:32.814589 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:11:32.814869 (+ 280us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:11:32.814877 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:11:32.815875 (+ 998us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:11:32.815882 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:11:32.817766 (+ 1884us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:11:32.817777 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:11:32.818926 (+ 1149us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:11:32.818932 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:11:32.819126 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:11:32.819176 (+ 50us) client_negotiation.cc:769] Sending connection context
0928 02:11:32.819263 (+ 87us) client_negotiation.cc:240] Negotiation successful
0928 02:11:32.819396 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":545,"thread_start_us":411,"threads_started":1}
I20260928 02:11:32.819710 14547 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:11:32.813641 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.235.65:58391 (local address 127.6.235.66:42855)
0928 02:11:32.813896 (+ 255us) server_negotiation.cc:206] Beginning negotiation
0928 02:11:32.813902 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 02:11:32.814173 (+ 271us) server_negotiation.cc:407] Connection header received
0928 02:11:32.814342 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:11:32.814347 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:11:32.814436 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:11:32.814539 (+ 103us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:11:32.815052 (+ 513us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:11:32.815730 (+ 678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:11:32.817914 (+ 2184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:11:32.818707 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:11:32.818888 (+ 181us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:11:32.818964 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0928 02:11:32.819324 (+ 360us) server_negotiation.cc:299] Negotiation successful
0928 02:11:32.819459 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":113}
I20260928 02:11:32.820096 14471 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "32ea9599979a4296accb4041d4587d45" candidate_uuid: "db7477740b844e9c8e8d14f8724f78d3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9dae072a16c44f1a8e4fe720ecff93cc" is_pre_election: true
I20260928 02:11:32.820653 14471 raft_consensus.cc:2454] T 32ea9599979a4296accb4041d4587d45 P 9dae072a16c44f1a8e4fe720ecff93cc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate db7477740b844e9c8e8d14f8724f78d3 in term 0.
I20260928 02:11:32.820868 14330 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "32ea9599979a4296accb4041d4587d45" candidate_uuid: "db7477740b844e9c8e8d14f8724f78d3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9f8e4ca0aeac43aa8ab6f8ab9a4d6fe9" is_pre_election: true
I20260928 02:11:32.821389 14117 leader_election.cc:302] T 32ea9599979a4296accb4041d4587d45 P db7477740b844e9c8e8d14f8724f78d3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9dae072a16c44f1a8e4fe720ecff93cc, db7477740b844e9c8e8d14f8724f78d3; no voters:
I20260928 02:11:32.821457 14330 raft_consensus.cc:2454] T 32ea9599979a4296accb4041d4587d45 P 9f8e4ca0aeac43aa8ab6f8ab9a4d6fe9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate db7477740b844e9c8e8d14f8724f78d3 in term 0.
I20260928 02:11:32.821817 14557 raft_consensus.cc:2781] T 32ea9599979a4296accb4041d4587d45 P db7477740b844e9c8e8d14f8724f78d3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 02:11:32.821985 14557 raft_consensus.cc:482] T 32ea9599979a4296accb4041d4587d45 P db7477740b844e9c8e8d14f8724f78d3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 02:11:32.822089 14557 raft_consensus.cc:3037] T 32ea9599979a4296accb4041d4587d45 P db7477740b844e9c8e8d14f8724f78d3 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:11:32.824290 14557 raft_consensus.cc:504] T 32ea9599979a4296accb4041d4587d45 P db7477740b844e9c8e8d14f8724f78d3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dae072a16c44f1a8e4fe720ecff93cc" member_type: VOTER last_known_addr { host: "127.6.235.67" port: 40303 } } peers { permanent_uuid: "9f8e4ca0aeac43aa8ab6f8ab9a4d6fe9" member_type: VOTER last_known_addr { host: "127.6.235.66" port: 42855 } } peers { permanent_uuid: "db7477740b844e9c8e8d14f8724f78d3" member_type: VOTER last_known_addr { host: "127.6.235.65" port: 42113 } }
I20260928 02:11:32.824918 14557 leader_election.cc:288] T 32ea9599979a4296accb4041d4587d45 P db7477740b844e9c8e8d14f8724f78d3 [CANDIDATE]: Term 1 election: Requested vote from peers 9dae072a16c44f1a8e4fe720ecff93cc (127.6.235.67:40303), 9f8e4ca0aeac43aa8ab6f8ab9a4d6fe9 (127.6.235.66:42855)
I20260928 02:11:32.825323 14471 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "32ea9599979a4296accb4041d4587d45" candidate_uuid: "db7477740b844e9c8e8d14f8724f78d3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9dae072a16c44f1a8e4fe720ecff93cc"
I20260928 02:11:32.825528 14471 raft_consensus.cc:3037] T 32ea9599979a4296accb4041d4587d45 P 9dae072a16c44f1a8e4fe720ecff93cc [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:11:32.825495 14330 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "32ea9599979a4296accb4041d4587d45" candidate_uuid: "db7477740b844e9c8e8d14f8724f78d3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9f8e4ca0aeac43aa8ab6f8ab9a4d6fe9"
I20260928 02:11:32.825830 14330 raft_consensus.cc:3037] T 32ea9599979a4296accb4041d4587d45 P 9f8e4ca0aeac43aa8ab6f8ab9a4d6fe9 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:11:32.827673 14471 raft_consensus.cc:2454] T 32ea9599979a4296accb4041d4587d45 P 9dae072a16c44f1a8e4fe720ecff93cc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate db7477740b844e9c8e8d14f8724f78d3 in term 1.
I20260928 02:11:32.828085 14117 leader_election.cc:302] T 32ea9599979a4296accb4041d4587d45 P db7477740b844e9c8e8d14f8724f78d3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9dae072a16c44f1a8e4fe720ecff93cc, db7477740b844e9c8e8d14f8724f78d3; no voters:
I20260928 02:11:32.828366 14557 raft_consensus.cc:2781] T 32ea9599979a4296accb4041d4587d45 P db7477740b844e9c8e8d14f8724f78d3 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:11:32.828399 14330 raft_consensus.cc:2454] T 32ea9599979a4296accb4041d4587d45 P 9f8e4ca0aeac43aa8ab6f8ab9a4d6fe9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate db7477740b844e9c8e8d14f8724f78d3 in term 1.
I20260928 02:11:32.828940 14557 raft_consensus.cc:686] T 32ea9599979a4296accb4041d4587d45 P db7477740b844e9c8e8d14f8724f78d3 [term 1 LEADER]: Becoming Leader. State: Replica: db7477740b844e9c8e8d14f8724f78d3, State: Running, Role: LEADER
I20260928 02:11:32.829504 14557 consensus_queue.cc:237] T 32ea9599979a4296accb4041d4587d45 P db7477740b844e9c8e8d14f8724f78d3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dae072a16c44f1a8e4fe720ecff93cc" member_type: VOTER last_known_addr { host: "127.6.235.67" port: 40303 } } peers { permanent_uuid: "9f8e4ca0aeac43aa8ab6f8ab9a4d6fe9" member_type: VOTER last_known_addr { host: "127.6.235.66" port: 42855 } } peers { permanent_uuid: "db7477740b844e9c8e8d14f8724f78d3" member_type: VOTER last_known_addr { host: "127.6.235.65" port: 42113 } }
I20260928 02:11:32.834702 13256 catalog_manager.cc:5495] T 32ea9599979a4296accb4041d4587d45 P db7477740b844e9c8e8d14f8724f78d3 reported cstate change: term changed from 0 to 1, leader changed from <none> to db7477740b844e9c8e8d14f8724f78d3 (127.6.235.65). New cstate: current_term: 1 leader_uuid: "db7477740b844e9c8e8d14f8724f78d3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dae072a16c44f1a8e4fe720ecff93cc" member_type: VOTER last_known_addr { host: "127.6.235.67" port: 40303 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9f8e4ca0aeac43aa8ab6f8ab9a4d6fe9" member_type: VOTER last_known_addr { host: "127.6.235.66" port: 42855 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "db7477740b844e9c8e8d14f8724f78d3" member_type: VOTER last_known_addr { host: "127.6.235.65" port: 42113 } health_report { overall_health: HEALTHY } } }
I20260928 02:11:32.873739 14549 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:11:32.867860 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36182 (local address 127.6.235.65:42113)
0928 02:11:32.868114 (+ 254us) server_negotiation.cc:206] Beginning negotiation
0928 02:11:32.868120 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 02:11:32.868280 (+ 160us) server_negotiation.cc:407] Connection header received
0928 02:11:32.868543 (+ 263us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:11:32.868549 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:11:32.868637 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:11:32.868803 (+ 166us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 02:11:32.869213 (+ 410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:11:32.869985 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:11:32.871401 (+ 1416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:11:32.871825 (+ 424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:11:32.871969 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:11:32.872488 (+ 519us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 02:11:32.873125 (+ 637us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 02:11:32.873287 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0928 02:11:32.873367 (+ 80us) server_negotiation.cc:299] Negotiation successful
0928 02:11:32.873451 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":119}
I20260928 02:11:32.882483 14471 raft_consensus.cc:1261] T 32ea9599979a4296accb4041d4587d45 P 9dae072a16c44f1a8e4fe720ecff93cc [term 1 FOLLOWER]: Refusing update from remote peer db7477740b844e9c8e8d14f8724f78d3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 02:11:32.882483 14330 raft_consensus.cc:1261] T 32ea9599979a4296accb4041d4587d45 P 9f8e4ca0aeac43aa8ab6f8ab9a4d6fe9 [term 1 FOLLOWER]: Refusing update from remote peer db7477740b844e9c8e8d14f8724f78d3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 02:11:32.883543 14561 consensus_queue.cc:1035] T 32ea9599979a4296accb4041d4587d45 P db7477740b844e9c8e8d14f8724f78d3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9f8e4ca0aeac43aa8ab6f8ab9a4d6fe9" member_type: VOTER last_known_addr { host: "127.6.235.66" port: 42855 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 02:11:32.883963 14557 consensus_queue.cc:1035] T 32ea9599979a4296accb4041d4587d45 P db7477740b844e9c8e8d14f8724f78d3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9dae072a16c44f1a8e4fe720ecff93cc" member_type: VOTER last_known_addr { host: "127.6.235.67" port: 40303 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 02:11:32.916373 13257 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42958:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 02:11:32.916661 13257 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42958:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 02:11:32.924391 13257 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 11adc61886584956adfdac31047373c1: Sending DeleteTablet for 3 replicas of tablet 32ea9599979a4296accb4041d4587d45
I20260928 02:11:32.925633 14451 tablet_service.cc:1514] Processing DeleteTablet for tablet 32ea9599979a4296accb4041d4587d45 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:11:32 UTC) from {username='oryx', principal='oryx/127.6.235.126@KRBTEST.COM'} at 127.0.0.1:50010
I20260928 02:11:32.927718 14310 tablet_service.cc:1514] Processing DeleteTablet for tablet 32ea9599979a4296accb4041d4587d45 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:11:32 UTC) from {username='oryx', principal='oryx/127.6.235.126@KRBTEST.COM'} at 127.0.0.1:50108
I20260928 02:11:32.927887 14574 tablet_replica.cc:330] T 32ea9599979a4296accb4041d4587d45 P 9dae072a16c44f1a8e4fe720ecff93cc: stopping tablet replica
I20260928 02:11:32.928231 14162 tablet_service.cc:1514] Processing DeleteTablet for tablet 32ea9599979a4296accb4041d4587d45 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:11:32 UTC) from {username='oryx', principal='oryx/127.6.235.126@KRBTEST.COM'} at 127.0.0.1:36176
I20260928 02:11:32.930905 14574 raft_consensus.cc:2229] T 32ea9599979a4296accb4041d4587d45 P 9dae072a16c44f1a8e4fe720ecff93cc [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 02:11:32.931442 14574 raft_consensus.cc:2258] T 32ea9599979a4296accb4041d4587d45 P 9dae072a16c44f1a8e4fe720ecff93cc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 02:11:32.932188 14576 tablet_replica.cc:330] T 32ea9599979a4296accb4041d4587d45 P db7477740b844e9c8e8d14f8724f78d3: stopping tablet replica
I20260928 02:11:32.932691 14576 raft_consensus.cc:2229] T 32ea9599979a4296accb4041d4587d45 P db7477740b844e9c8e8d14f8724f78d3 [term 1 LEADER]: Raft consensus shutting down.
I20260928 02:11:32.933463 14576 raft_consensus.cc:2258] T 32ea9599979a4296accb4041d4587d45 P db7477740b844e9c8e8d14f8724f78d3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 02:11:32.933692 14574 ts_tablet_manager.cc:1905] T 32ea9599979a4296accb4041d4587d45 P 9dae072a16c44f1a8e4fe720ecff93cc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 02:11:32.935757 14576 ts_tablet_manager.cc:1905] T 32ea9599979a4296accb4041d4587d45 P db7477740b844e9c8e8d14f8724f78d3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 02:11:32.939134 14575 tablet_replica.cc:330] T 32ea9599979a4296accb4041d4587d45 P 9f8e4ca0aeac43aa8ab6f8ab9a4d6fe9: stopping tablet replica
I20260928 02:11:32.939777 14575 raft_consensus.cc:2229] T 32ea9599979a4296accb4041d4587d45 P 9f8e4ca0aeac43aa8ab6f8ab9a4d6fe9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 02:11:32.940424 14575 raft_consensus.cc:2258] T 32ea9599979a4296accb4041d4587d45 P 9f8e4ca0aeac43aa8ab6f8ab9a4d6fe9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 02:11:32.941352 14574 ts_tablet_manager.cc:1918] T 32ea9599979a4296accb4041d4587d45 P 9dae072a16c44f1a8e4fe720ecff93cc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 02:11:32.941537 14574 log.cc:1196] T 32ea9599979a4296accb4041d4587d45 P 9dae072a16c44f1a8e4fe720ecff93cc: Deleting WAL directory at /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790561432251112-7085-0/minicluster-data/ts-2/wal/wals/32ea9599979a4296accb4041d4587d45
I20260928 02:11:32.942588 14574 ts_tablet_manager.cc:1939] T 32ea9599979a4296accb4041d4587d45 P 9dae072a16c44f1a8e4fe720ecff93cc: Deleting consensus metadata
I20260928 02:11:32.943239 14575 ts_tablet_manager.cc:1905] T 32ea9599979a4296accb4041d4587d45 P 9f8e4ca0aeac43aa8ab6f8ab9a4d6fe9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 02:11:32.943897 14576 ts_tablet_manager.cc:1918] T 32ea9599979a4296accb4041d4587d45 P db7477740b844e9c8e8d14f8724f78d3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 02:11:32.944123 14576 log.cc:1196] T 32ea9599979a4296accb4041d4587d45 P db7477740b844e9c8e8d14f8724f78d3: Deleting WAL directory at /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790561432251112-7085-0/minicluster-data/ts-0/wal/wals/32ea9599979a4296accb4041d4587d45
I20260928 02:11:32.944451 13243 catalog_manager.cc:4841] TS 9dae072a16c44f1a8e4fe720ecff93cc (127.6.235.67:40303): tablet 32ea9599979a4296accb4041d4587d45 (table test-table [id=975490bb04b249b69d5af592326250be]) successfully deleted
I20260928 02:11:32.945322 14576 ts_tablet_manager.cc:1939] T 32ea9599979a4296accb4041d4587d45 P db7477740b844e9c8e8d14f8724f78d3: Deleting consensus metadata
I20260928 02:11:32.946875 13242 catalog_manager.cc:4841] TS db7477740b844e9c8e8d14f8724f78d3 (127.6.235.65:42113): tablet 32ea9599979a4296accb4041d4587d45 (table test-table [id=975490bb04b249b69d5af592326250be]) successfully deleted
I20260928 02:11:32.950605 14575 ts_tablet_manager.cc:1918] T 32ea9599979a4296accb4041d4587d45 P 9f8e4ca0aeac43aa8ab6f8ab9a4d6fe9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 02:11:32.950764 14575 log.cc:1196] T 32ea9599979a4296accb4041d4587d45 P 9f8e4ca0aeac43aa8ab6f8ab9a4d6fe9: Deleting WAL directory at /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790561432251112-7085-0/minicluster-data/ts-1/wal/wals/32ea9599979a4296accb4041d4587d45
I20260928 02:11:32.951021 7085 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskTlaUuc/build/asan/bin/kudu pid 13225
I20260928 02:11:32.951318 14575 ts_tablet_manager.cc:1939] T 32ea9599979a4296accb4041d4587d45 P 9f8e4ca0aeac43aa8ab6f8ab9a4d6fe9: Deleting consensus metadata
I20260928 02:11:32.952179 13287 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 02:11:32.953930 13243 catalog_manager.cc:4841] TS 9f8e4ca0aeac43aa8ab6f8ab9a4d6fe9 (127.6.235.66:42855): tablet 32ea9599979a4296accb4041d4587d45 (table test-table [id=975490bb04b249b69d5af592326250be]) successfully deleted
I20260928 02:11:32.965341 14533 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:11:32.930723 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60178 (local address 127.6.235.125:41279)
0928 02:11:32.930912 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0928 02:11:32.930920 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 02:11:32.949553 (+ 18633us) server_negotiation.cc:407] Connection header received
0928 02:11:32.949992 (+ 439us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:11:32.949997 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:11:32.950089 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:11:32.950200 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:11:32.952503 (+ 2303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:11:32.953653 (+ 1150us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:11:32.955140 (+ 1487us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:11:32.955706 (+ 566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:11:32.955859 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:11:32.959283 (+ 3424us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:11:32.959289 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:11:32.959295 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:11:32.959340 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:11:32.962482 (+ 3142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:11:32.963183 (+ 701us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:11:32.963189 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:11:32.963193 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:11:32.963334 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:11:32.963724 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:11:32.963731 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:11:32.963735 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:11:32.964018 (+ 283us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:11:32.964298 (+ 280us) server_negotiation.cc:1035] Waiting for connection context
0928 02:11:32.964610 (+ 312us) server_negotiation.cc:299] Negotiation successful
0928 02:11:32.964827 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260928 02:11:32.978547 14534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:11:32.949824 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41044 (local address 127.6.235.124:33005)
0928 02:11:32.950024 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0928 02:11:32.950029 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 02:11:32.965250 (+ 15221us) server_negotiation.cc:407] Connection header received
0928 02:11:32.965423 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:11:32.965440 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:11:32.965510 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:11:32.965620 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:11:32.967298 (+ 1678us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:11:32.968063 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:11:32.969695 (+ 1632us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:11:32.970116 (+ 421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:11:32.970250 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:11:32.972824 (+ 2574us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:11:32.972829 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:11:32.972832 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:11:32.972866 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:11:32.975701 (+ 2835us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:11:32.976368 (+ 667us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:11:32.976374 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:11:32.976377 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:11:32.976504 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:11:32.976964 (+ 460us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:11:32.976968 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:11:32.976970 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:11:32.977243 (+ 273us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:11:32.977503 (+ 260us) server_negotiation.cc:1035] Waiting for connection context
0928 02:11:32.977870 (+ 367us) server_negotiation.cc:299] Negotiation successful
0928 02:11:32.978164 (+ 294us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260928 02:11:33.100615 7085 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTlaUuc/build/asan/bin/kudu with pid 13225
I20260928 02:11:33.158334 14534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:11:33.142436 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41052 (local address 127.6.235.124:33005)
0928 02:11:33.142607 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0928 02:11:33.142615 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 02:11:33.142825 (+ 210us) server_negotiation.cc:407] Connection header received
0928 02:11:33.143026 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:11:33.143052 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:11:33.143134 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:11:33.143212 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:11:33.145368 (+ 2156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:11:33.146109 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:11:33.148088 (+ 1979us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:11:33.148746 (+ 658us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:11:33.148879 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:11:33.151613 (+ 2734us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:11:33.151620 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:11:33.151625 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:11:33.151667 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:11:33.155345 (+ 3678us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:11:33.156179 (+ 834us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:11:33.156187 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:11:33.156193 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:11:33.156350 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:11:33.156766 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:11:33.156772 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:11:33.156775 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:11:33.157172 (+ 397us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:11:33.157431 (+ 259us) server_negotiation.cc:1035] Waiting for connection context
0928 02:11:33.157804 (+ 373us) server_negotiation.cc:299] Negotiation successful
0928 02:11:33.158016 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260928 02:11:33.161111 14533 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:11:33.141140 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60186 (local address 127.6.235.125:41279)
0928 02:11:33.141384 (+ 244us) server_negotiation.cc:206] Beginning negotiation
0928 02:11:33.141394 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 02:11:33.142429 (+ 1035us) server_negotiation.cc:407] Connection header received
0928 02:11:33.142726 (+ 297us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:11:33.142732 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:11:33.142819 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:11:33.142929 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:11:33.144996 (+ 2067us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:11:33.146203 (+ 1207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:11:33.148993 (+ 2790us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:11:33.149617 (+ 624us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:11:33.149774 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:11:33.154643 (+ 4869us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:11:33.154648 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:11:33.154651 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:11:33.154699 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:11:33.157942 (+ 3243us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:11:33.158733 (+ 791us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:11:33.158739 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:11:33.158743 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:11:33.158934 (+ 191us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:11:33.159364 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:11:33.159369 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:11:33.159373 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:11:33.159868 (+ 495us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:11:33.160122 (+ 254us) server_negotiation.cc:1035] Waiting for connection context
0928 02:11:33.160467 (+ 345us) server_negotiation.cc:299] Negotiation successful
0928 02:11:33.160758 (+ 291us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260928 02:11:33.904453 14585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:11:33.903593 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.235.126:34853 (local address 127.6.235.65:36789)
0928 02:11:33.904109 (+ 516us) negotiation.cc:107] Waiting for socket to connect
0928 02:11:33.904268 (+ 159us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.235.126:34853: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":386,"thread_start_us":240,"threads_started":1}
W20260928 02:11:33.904942 14228 heartbeater.cc:644] Failed to heartbeat to 127.6.235.126:34853 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.6.235.126:34853: connect: Connection refused (error 111)
I20260928 02:11:33.907603 14586 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:11:33.906710 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.235.126:34853 (local address 127.6.235.67:43743)
0928 02:11:33.907208 (+ 498us) negotiation.cc:107] Waiting for socket to connect
0928 02:11:33.907351 (+ 143us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.235.126:34853: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":370,"thread_start_us":235,"threads_started":1}
W20260928 02:11:33.908133 14517 heartbeater.cc:644] Failed to heartbeat to 127.6.235.126:34853 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.6.235.126:34853: connect: Connection refused (error 111)
I20260928 02:11:33.908708 14587 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:11:33.907682 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.235.126:34853 (local address 127.6.235.66:56065)
0928 02:11:33.908358 (+ 676us) negotiation.cc:107] Waiting for socket to connect
0928 02:11:33.908487 (+ 129us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.235.126:34853: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":547,"thread_start_us":476,"threads_started":1}
W20260928 02:11:33.909602 14376 heartbeater.cc:644] Failed to heartbeat to 127.6.235.126:34853 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.6.235.126:34853: connect: Connection refused (error 111)
I20260928 02:11:34.424871 14588 raft_consensus.cc:482] T 00000000000000000000000000000000 P bafe67917b9a4b40a60409c753bf4b96 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 11adc61886584956adfdac31047373c1)
I20260928 02:11:34.425268 14588 raft_consensus.cc:504] T 00000000000000000000000000000000 P bafe67917b9a4b40a60409c753bf4b96 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "11adc61886584956adfdac31047373c1" member_type: VOTER last_known_addr { host: "127.6.235.126" port: 34853 } } peers { permanent_uuid: "bafe67917b9a4b40a60409c753bf4b96" member_type: VOTER last_known_addr { host: "127.6.235.125" port: 41279 } attrs { promote: false } } peers { permanent_uuid: "254f36e26f8d4ddf823cf81c7a5c2986" member_type: VOTER last_known_addr { host: "127.6.235.124" port: 33005 } attrs { promote: false } }
I20260928 02:11:34.426846 14588 leader_election.cc:288] T 00000000000000000000000000000000 P bafe67917b9a4b40a60409c753bf4b96 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 11adc61886584956adfdac31047373c1 (127.6.235.126:34853), 254f36e26f8d4ddf823cf81c7a5c2986 (127.6.235.124:33005)
I20260928 02:11:34.439693 14589 raft_consensus.cc:482] T 00000000000000000000000000000000 P 254f36e26f8d4ddf823cf81c7a5c2986 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 11adc61886584956adfdac31047373c1)
I20260928 02:11:34.440069 14589 raft_consensus.cc:504] T 00000000000000000000000000000000 P 254f36e26f8d4ddf823cf81c7a5c2986 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "11adc61886584956adfdac31047373c1" member_type: VOTER last_known_addr { host: "127.6.235.126" port: 34853 } } peers { permanent_uuid: "bafe67917b9a4b40a60409c753bf4b96" member_type: VOTER last_known_addr { host: "127.6.235.125" port: 41279 } attrs { promote: false } } peers { permanent_uuid: "254f36e26f8d4ddf823cf81c7a5c2986" member_type: VOTER last_known_addr { host: "127.6.235.124" port: 33005 } attrs { promote: false } }
I20260928 02:11:34.443881 14590 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:11:34.427117 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.235.126:34853 (local address 127.0.0.1:43404)
0928 02:11:34.443322 (+ 16205us) negotiation.cc:107] Waiting for socket to connect
0928 02:11:34.443530 (+ 208us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.235.126:34853: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":16079,"thread_start_us":11980,"threads_started":1}
W20260928 02:11:34.444195 13843 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.235.126:34853: connect: Connection refused (error 111)
I20260928 02:11:34.444296 14593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:11:34.443044 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.235.126:34853 (local address 127.0.0.1:43422)
0928 02:11:34.443761 (+ 717us) negotiation.cc:107] Waiting for socket to connect
0928 02:11:34.443980 (+ 219us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.235.126:34853: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":563,"thread_start_us":329,"threads_started":1}
I20260928 02:11:34.444682 14589 leader_election.cc:288] T 00000000000000000000000000000000 P 254f36e26f8d4ddf823cf81c7a5c2986 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 11adc61886584956adfdac31047373c1 (127.6.235.126:34853), bafe67917b9a4b40a60409c753bf4b96 (127.6.235.125:41279)
W20260928 02:11:34.444748 14031 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.235.126:34853: connect: Connection refused (error 111)
I20260928 02:11:34.446285 14590 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:11:34.445938 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.235.126:34853 (local address 127.0.0.1:43428)
0928 02:11:34.446084 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0928 02:11:34.446169 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.235.126:34853: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260928 02:11:34.446593 13843 leader_election.cc:334] T 00000000000000000000000000000000 P bafe67917b9a4b40a60409c753bf4b96 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 11adc61886584956adfdac31047373c1 (127.6.235.126:34853): Network error: Client connection negotiation failed: client connection to 127.6.235.126:34853: connect: Connection refused (error 111)
I20260928 02:11:34.447573 14597 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:11:34.446629 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.235.126:34853 (local address 127.0.0.1:43430)
0928 02:11:34.447218 (+ 589us) negotiation.cc:107] Waiting for socket to connect
0928 02:11:34.447346 (+ 128us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.235.126:34853: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":487,"thread_start_us":378,"threads_started":1}
W20260928 02:11:34.447845 14031 leader_election.cc:334] T 00000000000000000000000000000000 P 254f36e26f8d4ddf823cf81c7a5c2986 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 11adc61886584956adfdac31047373c1 (127.6.235.126:34853): Network error: Client connection negotiation failed: client connection to 127.6.235.126:34853: connect: Connection refused (error 111)
I20260928 02:11:34.449988 14592 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:11:34.427548 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41066 (local address 127.6.235.124:33005)
0928 02:11:34.440997 (+ 13449us) server_negotiation.cc:206] Beginning negotiation
0928 02:11:34.441003 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 02:11:34.441024 (+ 21us) server_negotiation.cc:407] Connection header received
0928 02:11:34.441100 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:11:34.441105 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:11:34.441194 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:11:34.441297 (+ 103us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:11:34.442948 (+ 1651us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:11:34.444851 (+ 1903us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:11:34.447707 (+ 2856us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:11:34.449057 (+ 1350us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:11:34.449212 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:11:34.449326 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0928 02:11:34.449622 (+ 296us) server_negotiation.cc:299] Negotiation successful
0928 02:11:34.449736 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13297,"thread_start_us":14557,"threads_started":1}
I20260928 02:11:34.450073 14591 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:11:34.427103 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.235.124:33005 (local address 127.0.0.1:41066)
0928 02:11:34.439428 (+ 12325us) negotiation.cc:107] Waiting for socket to connect
0928 02:11:34.439456 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0928 02:11:34.439639 (+ 183us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:11:34.441511 (+ 1872us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:11:34.441516 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:11:34.441546 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:11:34.442692 (+ 1146us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:11:34.442722 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:11:34.445125 (+ 2403us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:11:34.445134 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:11:34.447522 (+ 2388us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:11:34.447541 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:11:34.449215 (+ 1674us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:11:34.449220 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:11:34.449437 (+ 217us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:11:34.449482 (+ 45us) client_negotiation.cc:769] Sending connection context
0928 02:11:34.449592 (+ 110us) client_negotiation.cc:240] Negotiation successful
0928 02:11:34.449752 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12174,"thread_start_us":15958,"threads_started":1}
I20260928 02:11:34.450778 14055 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "bafe67917b9a4b40a60409c753bf4b96" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "254f36e26f8d4ddf823cf81c7a5c2986" is_pre_election: true
I20260928 02:11:34.451122 14055 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 254f36e26f8d4ddf823cf81c7a5c2986 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bafe67917b9a4b40a60409c753bf4b96 in term 1.
I20260928 02:11:34.451607 13844 leader_election.cc:302] T 00000000000000000000000000000000 P bafe67917b9a4b40a60409c753bf4b96 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 254f36e26f8d4ddf823cf81c7a5c2986, bafe67917b9a4b40a60409c753bf4b96; no voters: 11adc61886584956adfdac31047373c1
I20260928 02:11:34.452015 14588 raft_consensus.cc:2781] T 00000000000000000000000000000000 P bafe67917b9a4b40a60409c753bf4b96 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260928 02:11:34.452153 14588 raft_consensus.cc:482] T 00000000000000000000000000000000 P bafe67917b9a4b40a60409c753bf4b96 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 11adc61886584956adfdac31047373c1)
I20260928 02:11:34.452265 14588 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bafe67917b9a4b40a60409c753bf4b96 [term 1 FOLLOWER]: Advancing to term 2
I20260928 02:11:34.453869 14593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:11:34.445356 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.235.125:41279 (local address 127.0.0.1:60190)
0928 02:11:34.446537 (+ 1181us) negotiation.cc:107] Waiting for socket to connect
0928 02:11:34.446556 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0928 02:11:34.446763 (+ 207us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:11:34.447409 (+ 646us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:11:34.447423 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:11:34.447474 (+ 51us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:11:34.447813 (+ 339us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:11:34.447833 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:11:34.449383 (+ 1550us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:11:34.449389 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:11:34.451365 (+ 1976us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:11:34.451379 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:11:34.452947 (+ 1568us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:11:34.452953 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:11:34.453186 (+ 233us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:11:34.453211 (+ 25us) client_negotiation.cc:769] Sending connection context
0928 02:11:34.453354 (+ 143us) client_negotiation.cc:240] Negotiation successful
0928 02:11:34.453502 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":211}
I20260928 02:11:34.453902 14594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:11:34.443945 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60190 (local address 127.6.235.125:41279)
0928 02:11:34.444573 (+ 628us) server_negotiation.cc:206] Beginning negotiation
0928 02:11:34.444580 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 02:11:34.446633 (+ 2053us) server_negotiation.cc:407] Connection header received
0928 02:11:34.446927 (+ 294us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:11:34.446934 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:11:34.447028 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:11:34.447104 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:11:34.448017 (+ 913us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:11:34.449236 (+ 1219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:11:34.451523 (+ 2287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:11:34.452750 (+ 1227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:11:34.452900 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:11:34.453039 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0928 02:11:34.453382 (+ 343us) server_negotiation.cc:299] Negotiation successful
0928 02:11:34.453515 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":495,"thread_start_us":260,"threads_started":1}
I20260928 02:11:34.454555 13867 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "254f36e26f8d4ddf823cf81c7a5c2986" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "bafe67917b9a4b40a60409c753bf4b96" is_pre_election: true
I20260928 02:11:34.455235 14588 raft_consensus.cc:504] T 00000000000000000000000000000000 P bafe67917b9a4b40a60409c753bf4b96 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "11adc61886584956adfdac31047373c1" member_type: VOTER last_known_addr { host: "127.6.235.126" port: 34853 } } peers { permanent_uuid: "bafe67917b9a4b40a60409c753bf4b96" member_type: VOTER last_known_addr { host: "127.6.235.125" port: 41279 } attrs { promote: false } } peers { permanent_uuid: "254f36e26f8d4ddf823cf81c7a5c2986" member_type: VOTER last_known_addr { host: "127.6.235.124" port: 33005 } attrs { promote: false } }
I20260928 02:11:34.455659 13867 raft_consensus.cc:2379] T 00000000000000000000000000000000 P bafe67917b9a4b40a60409c753bf4b96 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 254f36e26f8d4ddf823cf81c7a5c2986 in current term 2: Already voted for candidate bafe67917b9a4b40a60409c753bf4b96 in this term.
I20260928 02:11:34.456060 14588 leader_election.cc:288] T 00000000000000000000000000000000 P bafe67917b9a4b40a60409c753bf4b96 [CANDIDATE]: Term 2 election: Requested vote from peers 11adc61886584956adfdac31047373c1 (127.6.235.126:34853), 254f36e26f8d4ddf823cf81c7a5c2986 (127.6.235.124:33005)
I20260928 02:11:34.456647 14591 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:11:34.456160 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.235.126:34853 (local address 127.0.0.1:43442)
0928 02:11:34.456382 (+ 222us) negotiation.cc:107] Waiting for socket to connect
0928 02:11:34.456491 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.235.126:34853: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":91}
I20260928 02:11:34.456748 14055 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "bafe67917b9a4b40a60409c753bf4b96" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "254f36e26f8d4ddf823cf81c7a5c2986"
I20260928 02:11:34.457026 14055 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 254f36e26f8d4ddf823cf81c7a5c2986 [term 1 FOLLOWER]: Advancing to term 2
I20260928 02:11:34.457338 14032 leader_election.cc:302] T 00000000000000000000000000000000 P 254f36e26f8d4ddf823cf81c7a5c2986 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 254f36e26f8d4ddf823cf81c7a5c2986; no voters: 11adc61886584956adfdac31047373c1, bafe67917b9a4b40a60409c753bf4b96
I20260928 02:11:34.458166 14591 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:11:34.457671 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.235.126:34853 (local address 127.0.0.1:43450)
0928 02:11:34.457963 (+ 292us) negotiation.cc:107] Waiting for socket to connect
0928 02:11:34.458042 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.235.126:34853: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":211}
W20260928 02:11:34.458344 13843 leader_election.cc:334] T 00000000000000000000000000000000 P bafe67917b9a4b40a60409c753bf4b96 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 11adc61886584956adfdac31047373c1 (127.6.235.126:34853): Network error: Client connection negotiation failed: client connection to 127.6.235.126:34853: connect: Connection refused (error 111)
I20260928 02:11:34.459685 14055 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 254f36e26f8d4ddf823cf81c7a5c2986 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bafe67917b9a4b40a60409c753bf4b96 in term 2.
I20260928 02:11:34.459914 14589 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 254f36e26f8d4ddf823cf81c7a5c2986 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260928 02:11:34.460307 13844 leader_election.cc:302] T 00000000000000000000000000000000 P bafe67917b9a4b40a60409c753bf4b96 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 254f36e26f8d4ddf823cf81c7a5c2986, bafe67917b9a4b40a60409c753bf4b96; no voters: 11adc61886584956adfdac31047373c1
I20260928 02:11:34.460625 14588 raft_consensus.cc:2781] T 00000000000000000000000000000000 P bafe67917b9a4b40a60409c753bf4b96 [term 2 FOLLOWER]: Leader election won for term 2
I20260928 02:11:34.461112 14588 raft_consensus.cc:686] T 00000000000000000000000000000000 P bafe67917b9a4b40a60409c753bf4b96 [term 2 LEADER]: Becoming Leader. State: Replica: bafe67917b9a4b40a60409c753bf4b96, State: Running, Role: LEADER
I20260928 02:11:34.461542 14588 consensus_queue.cc:237] T 00000000000000000000000000000000 P bafe67917b9a4b40a60409c753bf4b96 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 12, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "11adc61886584956adfdac31047373c1" member_type: VOTER last_known_addr { host: "127.6.235.126" port: 34853 } } peers { permanent_uuid: "bafe67917b9a4b40a60409c753bf4b96" member_type: VOTER last_known_addr { host: "127.6.235.125" port: 41279 } attrs { promote: false } } peers { permanent_uuid: "254f36e26f8d4ddf823cf81c7a5c2986" member_type: VOTER last_known_addr { host: "127.6.235.124" port: 33005 } attrs { promote: false } }
I20260928 02:11:34.464023 14598 sys_catalog.cc:455] T 00000000000000000000000000000000 P bafe67917b9a4b40a60409c753bf4b96 [sys.catalog]: SysCatalogTable state changed. Reason: New leader bafe67917b9a4b40a60409c753bf4b96. Latest consensus state: current_term: 2 leader_uuid: "bafe67917b9a4b40a60409c753bf4b96" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "11adc61886584956adfdac31047373c1" member_type: VOTER last_known_addr { host: "127.6.235.126" port: 34853 } } peers { permanent_uuid: "bafe67917b9a4b40a60409c753bf4b96" member_type: VOTER last_known_addr { host: "127.6.235.125" port: 41279 } attrs { promote: false } } peers { permanent_uuid: "254f36e26f8d4ddf823cf81c7a5c2986" member_type: VOTER last_known_addr { host: "127.6.235.124" port: 33005 } attrs { promote: false } } }
I20260928 02:11:34.464339 14598 sys_catalog.cc:458] T 00000000000000000000000000000000 P bafe67917b9a4b40a60409c753bf4b96 [sys.catalog]: This master's current role is: LEADER
I20260928 02:11:34.465085 14600 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 02:11:34.468667 14600 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 02:11:34.469537 14600 catalog_manager.cc:1259] Loaded cluster ID: 33ac180e627642b2a4850f9fab0c8be0
I20260928 02:11:34.469617 14600 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 02:11:34.470741 14600 catalog_manager.cc:1504] Loading token signing keys...
I20260928 02:11:34.471474 14600 catalog_manager.cc:5879] T 00000000000000000000000000000000 P bafe67917b9a4b40a60409c753bf4b96: Loaded TSK: 0
I20260928 02:11:34.472301 14600 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 02:11:34.500862 13857 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60186:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 02:11:34.508208 13857 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 02:11:34.512930 14591 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:11:34.512420 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.235.126:34853 (local address 127.0.0.1:43470)
0928 02:11:34.512599 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0928 02:11:34.512730 (+ 131us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.235.126:34853: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
I20260928 02:11:34.513002 14055 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 254f36e26f8d4ddf823cf81c7a5c2986 [term 2 FOLLOWER]: Refusing update from remote peer bafe67917b9a4b40a60409c753bf4b96: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 2 index: 14. (index mismatch)
I20260928 02:11:34.513806 14598 consensus_queue.cc:1035] T 00000000000000000000000000000000 P bafe67917b9a4b40a60409c753bf4b96 [LEADER]: Connected to new peer: Peer: permanent_uuid: "254f36e26f8d4ddf823cf81c7a5c2986" member_type: VOTER last_known_addr { host: "127.6.235.124" port: 33005 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20260928 02:11:34.514217 14591 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:11:34.513820 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.235.126:34853 (local address 127.0.0.1:43478)
0928 02:11:34.513972 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0928 02:11:34.514043 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.235.126:34853: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260928 02:11:34.514410 13843 consensus_peers.cc:487] T 00000000000000000000000000000000 P bafe67917b9a4b40a60409c753bf4b96 -> Peer 11adc61886584956adfdac31047373c1 (127.6.235.126:34853): Couldn't send request to peer 11adc61886584956adfdac31047373c1. Status: Network error: Client connection negotiation failed: client connection to 127.6.235.126:34853: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260928 02:11:34.516745 14589 sys_catalog.cc:455] T 00000000000000000000000000000000 P 254f36e26f8d4ddf823cf81c7a5c2986 [sys.catalog]: SysCatalogTable state changed. Reason: New leader bafe67917b9a4b40a60409c753bf4b96. Latest consensus state: current_term: 2 leader_uuid: "bafe67917b9a4b40a60409c753bf4b96" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "11adc61886584956adfdac31047373c1" member_type: VOTER last_known_addr { host: "127.6.235.126" port: 34853 } } peers { permanent_uuid: "bafe67917b9a4b40a60409c753bf4b96" member_type: VOTER last_known_addr { host: "127.6.235.125" port: 41279 } attrs { promote: false } } peers { permanent_uuid: "254f36e26f8d4ddf823cf81c7a5c2986" member_type: VOTER last_known_addr { host: "127.6.235.124" port: 33005 } attrs { promote: false } } }
I20260928 02:11:34.517230 14589 sys_catalog.cc:458] T 00000000000000000000000000000000 P 254f36e26f8d4ddf823cf81c7a5c2986 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 02:11:34.518580 14588 sys_catalog.cc:455] T 00000000000000000000000000000000 P bafe67917b9a4b40a60409c753bf4b96 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 2 leader_uuid: "bafe67917b9a4b40a60409c753bf4b96" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "11adc61886584956adfdac31047373c1" member_type: VOTER last_known_addr { host: "127.6.235.126" port: 34853 } } peers { permanent_uuid: "bafe67917b9a4b40a60409c753bf4b96" member_type: VOTER last_known_addr { host: "127.6.235.125" port: 41279 } attrs { promote: false } } peers { permanent_uuid: "254f36e26f8d4ddf823cf81c7a5c2986" member_type: VOTER last_known_addr { host: "127.6.235.124" port: 33005 } attrs { promote: false } } }
I20260928 02:11:34.518958 14588 sys_catalog.cc:458] T 00000000000000000000000000000000 P bafe67917b9a4b40a60409c753bf4b96 [sys.catalog]: This master's current role is: LEADER
I20260928 02:11:34.519408 14589 sys_catalog.cc:455] T 00000000000000000000000000000000 P 254f36e26f8d4ddf823cf81c7a5c2986 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 2 leader_uuid: "bafe67917b9a4b40a60409c753bf4b96" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "11adc61886584956adfdac31047373c1" member_type: VOTER last_known_addr { host: "127.6.235.126" port: 34853 } } peers { permanent_uuid: "bafe67917b9a4b40a60409c753bf4b96" member_type: VOTER last_known_addr { host: "127.6.235.125" port: 41279 } attrs { promote: false } } peers { permanent_uuid: "254f36e26f8d4ddf823cf81c7a5c2986" member_type: VOTER last_known_addr { host: "127.6.235.124" port: 33005 } attrs { promote: false } } }
I20260928 02:11:34.519646 14589 sys_catalog.cc:458] T 00000000000000000000000000000000 P 254f36e26f8d4ddf823cf81c7a5c2986 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 02:11:34.535717 14590 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:11:34.528407 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.235.66:42855 (local address 127.0.0.1:50110)
0928 02:11:34.528623 (+ 216us) negotiation.cc:107] Waiting for socket to connect
0928 02:11:34.528645 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0928 02:11:34.528776 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:11:34.529470 (+ 694us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:11:34.529481 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:11:34.529535 (+ 54us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:11:34.529940 (+ 405us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:11:34.529954 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:11:34.531791 (+ 1837us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:11:34.531795 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:11:34.533305 (+ 1510us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:11:34.533319 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:11:34.534903 (+ 1584us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:11:34.534912 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:11:34.535089 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:11:34.535116 (+ 27us) client_negotiation.cc:769] Sending connection context
0928 02:11:34.535231 (+ 115us) client_negotiation.cc:240] Negotiation successful
0928 02:11:34.535407 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":99}
I20260928 02:11:34.535729 14608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:11:34.528532 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50110 (local address 127.6.235.66:42855)
0928 02:11:34.529120 (+ 588us) server_negotiation.cc:206] Beginning negotiation
0928 02:11:34.529127 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 02:11:34.529146 (+ 19us) server_negotiation.cc:407] Connection header received
0928 02:11:34.529217 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:11:34.529222 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:11:34.529294 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:11:34.530103 (+ 809us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:11:34.530303 (+ 200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:11:34.531639 (+ 1336us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:11:34.533487 (+ 1848us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:11:34.534691 (+ 1204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:11:34.534795 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:11:34.534867 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0928 02:11:34.535261 (+ 394us) server_negotiation.cc:299] Negotiation successful
0928 02:11:34.535407 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":475,"thread_start_us":296,"threads_started":1}
I20260928 02:11:34.536991 14310 tablet_service.cc:1467] Processing CreateTablet for tablet da9baf7d3625467bbad07bcdeb4af3c7 (DEFAULT_TABLE table=test-table [id=353a4fbbba284d50a376c6dec5c8e7e6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:11:34.537514 14310 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet da9baf7d3625467bbad07bcdeb4af3c7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:11:34.540119 14591 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:11:34.528047 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.235.67:40303 (local address 127.0.0.1:50016)
0928 02:11:34.528446 (+ 399us) negotiation.cc:107] Waiting for socket to connect
0928 02:11:34.528469 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0928 02:11:34.528583 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:11:34.534606 (+ 6023us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:11:34.534611 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:11:34.534636 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:11:34.534807 (+ 171us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:11:34.534820 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:11:34.536243 (+ 1423us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:11:34.536252 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:11:34.538170 (+ 1918us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:11:34.538183 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:11:34.539380 (+ 1197us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:11:34.539386 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:11:34.539569 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:11:34.539592 (+ 23us) client_negotiation.cc:769] Sending connection context
0928 02:11:34.539732 (+ 140us) client_negotiation.cc:240] Negotiation successful
0928 02:11:34.539830 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":286,"mutex_wait_us":25}
I20260928 02:11:34.540781 14607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:11:34.528176 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50016 (local address 127.6.235.67:40303)
0928 02:11:34.534115 (+ 5939us) server_negotiation.cc:206] Beginning negotiation
0928 02:11:34.534123 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 02:11:34.534153 (+ 30us) server_negotiation.cc:407] Connection header received
0928 02:11:34.534219 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:11:34.534224 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:11:34.534307 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:11:34.534416 (+ 109us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:11:34.534981 (+ 565us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:11:34.536066 (+ 1085us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:11:34.538347 (+ 2281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:11:34.539218 (+ 871us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:11:34.539370 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:11:34.539482 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0928 02:11:34.540350 (+ 868us) server_negotiation.cc:299] Negotiation successful
0928 02:11:34.540491 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5821,"thread_start_us":5727,"threads_started":1}
I20260928 02:11:34.541451 14610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:11:34.529230 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36186 (local address 127.6.235.65:42113)
0928 02:11:34.533383 (+ 4153us) server_negotiation.cc:206] Beginning negotiation
0928 02:11:34.533390 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 02:11:34.533411 (+ 21us) server_negotiation.cc:407] Connection header received
0928 02:11:34.533480 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:11:34.533484 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:11:34.533575 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:11:34.533670 (+ 95us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:11:34.535622 (+ 1952us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:11:34.536565 (+ 943us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:11:34.538577 (+ 2012us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:11:34.539836 (+ 1259us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:11:34.539973 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:11:34.540066 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0928 02:11:34.541081 (+ 1015us) server_negotiation.cc:299] Negotiation successful
0928 02:11:34.541228 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4027,"thread_start_us":3889,"threads_started":1}
I20260928 02:11:34.540858 14609 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:11:34.529086 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.235.65:42113 (local address 127.0.0.1:36186)
0928 02:11:34.529717 (+ 631us) negotiation.cc:107] Waiting for socket to connect
0928 02:11:34.529734 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0928 02:11:34.529842 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:11:34.534960 (+ 5118us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:11:34.534965 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:11:34.534995 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:11:34.535426 (+ 431us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:11:34.535439 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:11:34.536749 (+ 1310us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:11:34.536757 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:11:34.538405 (+ 1648us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:11:34.538418 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:11:34.540051 (+ 1633us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:11:34.540057 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:11:34.540236 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:11:34.540271 (+ 35us) client_negotiation.cc:769] Sending connection context
0928 02:11:34.540359 (+ 88us) client_negotiation.cc:240] Negotiation successful
0928 02:11:34.540485 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":485,"mutex_wait_us":20,"thread_start_us":332,"threads_started":1}
I20260928 02:11:34.541574 14451 tablet_service.cc:1467] Processing CreateTablet for tablet da9baf7d3625467bbad07bcdeb4af3c7 (DEFAULT_TABLE table=test-table [id=353a4fbbba284d50a376c6dec5c8e7e6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:11:34.541994 14451 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet da9baf7d3625467bbad07bcdeb4af3c7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:11:34.542289 14162 tablet_service.cc:1467] Processing CreateTablet for tablet da9baf7d3625467bbad07bcdeb4af3c7 (DEFAULT_TABLE table=test-table [id=353a4fbbba284d50a376c6dec5c8e7e6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:11:34.542663 14162 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet da9baf7d3625467bbad07bcdeb4af3c7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:11:34.544238 14611 tablet_bootstrap.cc:492] T da9baf7d3625467bbad07bcdeb4af3c7 P 9f8e4ca0aeac43aa8ab6f8ab9a4d6fe9: Bootstrap starting.
I20260928 02:11:34.546044 14611 tablet_bootstrap.cc:654] T da9baf7d3625467bbad07bcdeb4af3c7 P 9f8e4ca0aeac43aa8ab6f8ab9a4d6fe9: Neither blocks nor log segments found. Creating new log.
I20260928 02:11:34.548282 14611 tablet_bootstrap.cc:492] T da9baf7d3625467bbad07bcdeb4af3c7 P 9f8e4ca0aeac43aa8ab6f8ab9a4d6fe9: No bootstrap required, opened a new log
I20260928 02:11:34.548422 14611 ts_tablet_manager.cc:1397] T da9baf7d3625467bbad07bcdeb4af3c7 P 9f8e4ca0aeac43aa8ab6f8ab9a4d6fe9: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20260928 02:11:34.549239 14613 tablet_bootstrap.cc:492] T da9baf7d3625467bbad07bcdeb4af3c7 P 9dae072a16c44f1a8e4fe720ecff93cc: Bootstrap starting.
I20260928 02:11:34.549460 14611 raft_consensus.cc:348] T da9baf7d3625467bbad07bcdeb4af3c7 P 9f8e4ca0aeac43aa8ab6f8ab9a4d6fe9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dae072a16c44f1a8e4fe720ecff93cc" member_type: VOTER last_known_addr { host: "127.6.235.67" port: 40303 } } peers { permanent_uuid: "9f8e4ca0aeac43aa8ab6f8ab9a4d6fe9" member_type: VOTER last_known_addr { host: "127.6.235.66" port: 42855 } } peers { permanent_uuid: "db7477740b844e9c8e8d14f8724f78d3" member_type: VOTER last_known_addr { host: "127.6.235.65" port: 42113 } }
I20260928 02:11:34.549819 14611 raft_consensus.cc:374] T da9baf7d3625467bbad07bcdeb4af3c7 P 9f8e4ca0aeac43aa8ab6f8ab9a4d6fe9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:11:34.549930 14611 raft_consensus.cc:729] T da9baf7d3625467bbad07bcdeb4af3c7 P 9f8e4ca0aeac43aa8ab6f8ab9a4d6fe9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9f8e4ca0aeac43aa8ab6f8ab9a4d6fe9, State: Initialized, Role: FOLLOWER
I20260928 02:11:34.550266 14611 consensus_queue.cc:260] T da9baf7d3625467bbad07bcdeb4af3c7 P 9f8e4ca0aeac43aa8ab6f8ab9a4d6fe9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dae072a16c44f1a8e4fe720ecff93cc" member_type: VOTER last_known_addr { host: "127.6.235.67" port: 40303 } } peers { permanent_uuid: "9f8e4ca0aeac43aa8ab6f8ab9a4d6fe9" member_type: VOTER last_known_addr { host: "127.6.235.66" port: 42855 } } peers { permanent_uuid: "db7477740b844e9c8e8d14f8724f78d3" member_type: VOTER last_known_addr { host: "127.6.235.65" port: 42113 } }
I20260928 02:11:34.551461 14613 tablet_bootstrap.cc:654] T da9baf7d3625467bbad07bcdeb4af3c7 P 9dae072a16c44f1a8e4fe720ecff93cc: Neither blocks nor log segments found. Creating new log.
I20260928 02:11:34.553602 14613 tablet_bootstrap.cc:492] T da9baf7d3625467bbad07bcdeb4af3c7 P 9dae072a16c44f1a8e4fe720ecff93cc: No bootstrap required, opened a new log
I20260928 02:11:34.553769 14613 ts_tablet_manager.cc:1397] T da9baf7d3625467bbad07bcdeb4af3c7 P 9dae072a16c44f1a8e4fe720ecff93cc: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.003s
I20260928 02:11:34.553874 14614 tablet_bootstrap.cc:492] T da9baf7d3625467bbad07bcdeb4af3c7 P db7477740b844e9c8e8d14f8724f78d3: Bootstrap starting.
I20260928 02:11:34.555677 14613 raft_consensus.cc:348] T da9baf7d3625467bbad07bcdeb4af3c7 P 9dae072a16c44f1a8e4fe720ecff93cc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dae072a16c44f1a8e4fe720ecff93cc" member_type: VOTER last_known_addr { host: "127.6.235.67" port: 40303 } } peers { permanent_uuid: "9f8e4ca0aeac43aa8ab6f8ab9a4d6fe9" member_type: VOTER last_known_addr { host: "127.6.235.66" port: 42855 } } peers { permanent_uuid: "db7477740b844e9c8e8d14f8724f78d3" member_type: VOTER last_known_addr { host: "127.6.235.65" port: 42113 } }
I20260928 02:11:34.555951 14613 raft_consensus.cc:374] T da9baf7d3625467bbad07bcdeb4af3c7 P 9dae072a16c44f1a8e4fe720ecff93cc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:11:34.556205 14613 raft_consensus.cc:729] T da9baf7d3625467bbad07bcdeb4af3c7 P 9dae072a16c44f1a8e4fe720ecff93cc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9dae072a16c44f1a8e4fe720ecff93cc, State: Initialized, Role: FOLLOWER
I20260928 02:11:34.556499 14614 tablet_bootstrap.cc:654] T da9baf7d3625467bbad07bcdeb4af3c7 P db7477740b844e9c8e8d14f8724f78d3: Neither blocks nor log segments found. Creating new log.
I20260928 02:11:34.556514 14613 consensus_queue.cc:260] T da9baf7d3625467bbad07bcdeb4af3c7 P 9dae072a16c44f1a8e4fe720ecff93cc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dae072a16c44f1a8e4fe720ecff93cc" member_type: VOTER last_known_addr { host: "127.6.235.67" port: 40303 } } peers { permanent_uuid: "9f8e4ca0aeac43aa8ab6f8ab9a4d6fe9" member_type: VOTER last_known_addr { host: "127.6.235.66" port: 42855 } } peers { permanent_uuid: "db7477740b844e9c8e8d14f8724f78d3" member_type: VOTER last_known_addr { host: "127.6.235.65" port: 42113 } }
I20260928 02:11:34.563329 14611 ts_tablet_manager.cc:1428] T da9baf7d3625467bbad07bcdeb4af3c7 P 9f8e4ca0aeac43aa8ab6f8ab9a4d6fe9: Time spent starting tablet: real 0.015s user 0.003s sys 0.003s
I20260928 02:11:34.563695 14613 ts_tablet_manager.cc:1428] T da9baf7d3625467bbad07bcdeb4af3c7 P 9dae072a16c44f1a8e4fe720ecff93cc: Time spent starting tablet: real 0.010s user 0.000s sys 0.009s
I20260928 02:11:34.564041 14377 heartbeater.cc:503] Master 127.6.235.125:41279 was elected leader, sending a full tablet report...
I20260928 02:11:34.577834 14518 heartbeater.cc:503] Master 127.6.235.125:41279 was elected leader, sending a full tablet report...
I20260928 02:11:34.578189 14617 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:11:34.556398 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.235.126:34853 (local address 127.6.235.66:44173)
0928 02:11:34.577658 (+ 21260us) negotiation.cc:107] Waiting for socket to connect
0928 02:11:34.577842 (+ 184us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.235.126:34853: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":21126,"thread_start_us":20983,"threads_started":1}
I20260928 02:11:34.581003 14617 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:11:34.579006 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.235.126:34853 (local address 127.6.235.66:42593)
0928 02:11:34.580731 (+ 1725us) negotiation.cc:107] Waiting for socket to connect
0928 02:11:34.580806 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.235.126:34853: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1629}
I20260928 02:11:34.582870 14614 tablet_bootstrap.cc:492] T da9baf7d3625467bbad07bcdeb4af3c7 P db7477740b844e9c8e8d14f8724f78d3: No bootstrap required, opened a new log
I20260928 02:11:34.583083 14614 ts_tablet_manager.cc:1397] T da9baf7d3625467bbad07bcdeb4af3c7 P db7477740b844e9c8e8d14f8724f78d3: Time spent bootstrapping tablet: real 0.029s user 0.013s sys 0.013s
I20260928 02:11:34.584066 14620 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:11:34.568840 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.235.126:34853 (local address 127.6.235.67:43851)
0928 02:11:34.583625 (+ 14785us) negotiation.cc:107] Waiting for socket to connect
0928 02:11:34.583784 (+ 159us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.235.126:34853: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":14634,"thread_start_us":14466,"threads_started":1}
I20260928 02:11:34.584112 14614 raft_consensus.cc:348] T da9baf7d3625467bbad07bcdeb4af3c7 P db7477740b844e9c8e8d14f8724f78d3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dae072a16c44f1a8e4fe720ecff93cc" member_type: VOTER last_known_addr { host: "127.6.235.67" port: 40303 } } peers { permanent_uuid: "9f8e4ca0aeac43aa8ab6f8ab9a4d6fe9" member_type: VOTER last_known_addr { host: "127.6.235.66" port: 42855 } } peers { permanent_uuid: "db7477740b844e9c8e8d14f8724f78d3" member_type: VOTER last_known_addr { host: "127.6.235.65" port: 42113 } }
I20260928 02:11:34.584398 14614 raft_consensus.cc:374] T da9baf7d3625467bbad07bcdeb4af3c7 P db7477740b844e9c8e8d14f8724f78d3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:11:34.584501 14614 raft_consensus.cc:729] T da9baf7d3625467bbad07bcdeb4af3c7 P db7477740b844e9c8e8d14f8724f78d3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db7477740b844e9c8e8d14f8724f78d3, State: Initialized, Role: FOLLOWER
I20260928 02:11:34.584757 14614 consensus_queue.cc:260] T da9baf7d3625467bbad07bcdeb4af3c7 P db7477740b844e9c8e8d14f8724f78d3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dae072a16c44f1a8e4fe720ecff93cc" member_type: VOTER last_known_addr { host: "127.6.235.67" port: 40303 } } peers { permanent_uuid: "9f8e4ca0aeac43aa8ab6f8ab9a4d6fe9" member_type: VOTER last_known_addr { host: "127.6.235.66" port: 42855 } } peers { permanent_uuid: "db7477740b844e9c8e8d14f8724f78d3" member_type: VOTER last_known_addr { host: "127.6.235.65" port: 42113 } }
I20260928 02:11:34.585282 14620 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:11:34.584829 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.235.126:34853 (local address 127.6.235.67:57245)
0928 02:11:34.585028 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0928 02:11:34.585127 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.235.126:34853: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":79}
I20260928 02:11:34.585598 14614 ts_tablet_manager.cc:1428] T da9baf7d3625467bbad07bcdeb4af3c7 P db7477740b844e9c8e8d14f8724f78d3: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260928 02:11:34.587602 14622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:11:34.586228 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.235.126:34853 (local address 127.6.235.65:38007)
0928 02:11:34.586956 (+ 728us) negotiation.cc:107] Waiting for socket to connect
0928 02:11:34.587313 (+ 357us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.235.126:34853: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":614,"thread_start_us":486,"threads_started":1}
I20260928 02:11:34.587941 14232 heartbeater.cc:503] Master 127.6.235.125:41279 was elected leader, sending a full tablet report...
I20260928 02:11:34.589176 14623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:11:34.588193 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.235.126:34853 (local address 127.6.235.65:57761)
0928 02:11:34.588725 (+ 532us) negotiation.cc:107] Waiting for socket to connect
0928 02:11:34.588925 (+ 200us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.235.126:34853: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":404,"thread_start_us":281,"threads_started":1}
I20260928 02:11:34.645263 14618 raft_consensus.cc:482] T da9baf7d3625467bbad07bcdeb4af3c7 P 9dae072a16c44f1a8e4fe720ecff93cc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 02:11:34.645546 14618 raft_consensus.cc:504] T da9baf7d3625467bbad07bcdeb4af3c7 P 9dae072a16c44f1a8e4fe720ecff93cc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dae072a16c44f1a8e4fe720ecff93cc" member_type: VOTER last_known_addr { host: "127.6.235.67" port: 40303 } } peers { permanent_uuid: "9f8e4ca0aeac43aa8ab6f8ab9a4d6fe9" member_type: VOTER last_known_addr { host: "127.6.235.66" port: 42855 } } peers { permanent_uuid: "db7477740b844e9c8e8d14f8724f78d3" member_type: VOTER last_known_addr { host: "127.6.235.65" port: 42113 } }
I20260928 02:11:34.647776 14618 leader_election.cc:288] T da9baf7d3625467bbad07bcdeb4af3c7 P 9dae072a16c44f1a8e4fe720ecff93cc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9f8e4ca0aeac43aa8ab6f8ab9a4d6fe9 (127.6.235.66:42855), db7477740b844e9c8e8d14f8724f78d3 (127.6.235.65:42113)
I20260928 02:11:34.655009 14608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:11:34.648214 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.235.67:60583 (local address 127.6.235.66:42855)
0928 02:11:34.649023 (+ 809us) server_negotiation.cc:206] Beginning negotiation
0928 02:11:34.649030 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 02:11:34.649051 (+ 21us) server_negotiation.cc:407] Connection header received
0928 02:11:34.649129 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:11:34.649135 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:11:34.649215 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:11:34.649312 (+ 97us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:11:34.650010 (+ 698us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:11:34.651089 (+ 1079us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:11:34.653046 (+ 1957us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:11:34.653958 (+ 912us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:11:34.654132 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:11:34.654215 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0928 02:11:34.654542 (+ 327us) server_negotiation.cc:299] Negotiation successful
0928 02:11:34.654693 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":625}
I20260928 02:11:34.655040 14620 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:11:34.648042 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.235.66:42855 (local address 127.6.235.67:60583)
0928 02:11:34.648436 (+ 394us) negotiation.cc:107] Waiting for socket to connect
0928 02:11:34.648456 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0928 02:11:34.648628 (+ 172us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:11:34.649515 (+ 887us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:11:34.649520 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:11:34.649548 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:11:34.649819 (+ 271us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:11:34.649834 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:11:34.651224 (+ 1390us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:11:34.651230 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:11:34.652836 (+ 1606us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:11:34.652849 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:11:34.654179 (+ 1330us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:11:34.654185 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:11:34.654371 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:11:34.654396 (+ 25us) client_negotiation.cc:769] Sending connection context
0928 02:11:34.654515 (+ 119us) client_negotiation.cc:240] Negotiation successful
0928 02:11:34.654676 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":237,"mutex_wait_us":23}
I20260928 02:11:34.655900 14624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:11:34.648197 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.235.65:42113 (local address 127.6.235.67:36265)
0928 02:11:34.649216 (+ 1019us) negotiation.cc:107] Waiting for socket to connect
0928 02:11:34.649234 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0928 02:11:34.649429 (+ 195us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:11:34.649906 (+ 477us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:11:34.649911 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:11:34.649964 (+ 53us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:11:34.650293 (+ 329us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:11:34.650306 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:11:34.651347 (+ 1041us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:11:34.651350 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:11:34.652981 (+ 1631us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:11:34.652998 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:11:34.654193 (+ 1195us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:11:34.654199 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:11:34.655355 (+ 1156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:11:34.655410 (+ 55us) client_negotiation.cc:769] Sending connection context
0928 02:11:34.655507 (+ 97us) client_negotiation.cc:240] Negotiation successful
0928 02:11:34.655631 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":891,"mutex_wait_us":4,"thread_start_us":914,"threads_started":1}
I20260928 02:11:34.655957 14610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:11:34.648412 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.235.67:36265 (local address 127.6.235.65:42113)
0928 02:11:34.648635 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0928 02:11:34.648645 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 02:11:34.649320 (+ 675us) server_negotiation.cc:407] Connection header received
0928 02:11:34.649611 (+ 291us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:11:34.649619 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:11:34.649730 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:11:34.649894 (+ 164us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:11:34.650445 (+ 551us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:11:34.651209 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:11:34.653143 (+ 1934us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:11:34.653967 (+ 824us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:11:34.654158 (+ 191us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:11:34.654235 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0928 02:11:34.655547 (+ 1312us) server_negotiation.cc:299] Negotiation successful
0928 02:11:34.655697 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260928 02:11:34.655838 14330 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da9baf7d3625467bbad07bcdeb4af3c7" candidate_uuid: "9dae072a16c44f1a8e4fe720ecff93cc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9f8e4ca0aeac43aa8ab6f8ab9a4d6fe9" is_pre_election: true
I20260928 02:11:34.656106 14330 raft_consensus.cc:2454] T da9baf7d3625467bbad07bcdeb4af3c7 P 9f8e4ca0aeac43aa8ab6f8ab9a4d6fe9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9dae072a16c44f1a8e4fe720ecff93cc in term 0.
I20260928 02:11:34.656631 14406 leader_election.cc:302] T da9baf7d3625467bbad07bcdeb4af3c7 P 9dae072a16c44f1a8e4fe720ecff93cc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9dae072a16c44f1a8e4fe720ecff93cc, 9f8e4ca0aeac43aa8ab6f8ab9a4d6fe9; no voters:
I20260928 02:11:34.656960 14182 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da9baf7d3625467bbad07bcdeb4af3c7" candidate_uuid: "9dae072a16c44f1a8e4fe720ecff93cc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db7477740b844e9c8e8d14f8724f78d3" is_pre_election: true
I20260928 02:11:34.657233 14618 raft_consensus.cc:2781] T da9baf7d3625467bbad07bcdeb4af3c7 P 9dae072a16c44f1a8e4fe720ecff93cc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 02:11:34.657348 14618 raft_consensus.cc:482] T da9baf7d3625467bbad07bcdeb4af3c7 P 9dae072a16c44f1a8e4fe720ecff93cc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 02:11:34.657467 14618 raft_consensus.cc:3037] T da9baf7d3625467bbad07bcdeb4af3c7 P 9dae072a16c44f1a8e4fe720ecff93cc [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:11:34.657416 14182 raft_consensus.cc:2454] T da9baf7d3625467bbad07bcdeb4af3c7 P db7477740b844e9c8e8d14f8724f78d3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9dae072a16c44f1a8e4fe720ecff93cc in term 0.
I20260928 02:11:34.660001 14618 raft_consensus.cc:504] T da9baf7d3625467bbad07bcdeb4af3c7 P 9dae072a16c44f1a8e4fe720ecff93cc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dae072a16c44f1a8e4fe720ecff93cc" member_type: VOTER last_known_addr { host: "127.6.235.67" port: 40303 } } peers { permanent_uuid: "9f8e4ca0aeac43aa8ab6f8ab9a4d6fe9" member_type: VOTER last_known_addr { host: "127.6.235.66" port: 42855 } } peers { permanent_uuid: "db7477740b844e9c8e8d14f8724f78d3" member_type: VOTER last_known_addr { host: "127.6.235.65" port: 42113 } }
I20260928 02:11:34.660622 14618 leader_election.cc:288] T da9baf7d3625467bbad07bcdeb4af3c7 P 9dae072a16c44f1a8e4fe720ecff93cc [CANDIDATE]: Term 1 election: Requested vote from peers 9f8e4ca0aeac43aa8ab6f8ab9a4d6fe9 (127.6.235.66:42855), db7477740b844e9c8e8d14f8724f78d3 (127.6.235.65:42113)
I20260928 02:11:34.661180 14330 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da9baf7d3625467bbad07bcdeb4af3c7" candidate_uuid: "9dae072a16c44f1a8e4fe720ecff93cc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9f8e4ca0aeac43aa8ab6f8ab9a4d6fe9"
I20260928 02:11:34.661273 14182 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da9baf7d3625467bbad07bcdeb4af3c7" candidate_uuid: "9dae072a16c44f1a8e4fe720ecff93cc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db7477740b844e9c8e8d14f8724f78d3"
I20260928 02:11:34.661417 14330 raft_consensus.cc:3037] T da9baf7d3625467bbad07bcdeb4af3c7 P 9f8e4ca0aeac43aa8ab6f8ab9a4d6fe9 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:11:34.661466 14182 raft_consensus.cc:3037] T da9baf7d3625467bbad07bcdeb4af3c7 P db7477740b844e9c8e8d14f8724f78d3 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:11:34.664557 14330 raft_consensus.cc:2454] T da9baf7d3625467bbad07bcdeb4af3c7 P 9f8e4ca0aeac43aa8ab6f8ab9a4d6fe9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9dae072a16c44f1a8e4fe720ecff93cc in term 1.
I20260928 02:11:34.664556 14182 raft_consensus.cc:2454] T da9baf7d3625467bbad07bcdeb4af3c7 P db7477740b844e9c8e8d14f8724f78d3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9dae072a16c44f1a8e4fe720ecff93cc in term 1.
I20260928 02:11:34.665094 14406 leader_election.cc:302] T da9baf7d3625467bbad07bcdeb4af3c7 P 9dae072a16c44f1a8e4fe720ecff93cc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9dae072a16c44f1a8e4fe720ecff93cc, 9f8e4ca0aeac43aa8ab6f8ab9a4d6fe9; no voters:
I20260928 02:11:34.665472 14618 raft_consensus.cc:2781] T da9baf7d3625467bbad07bcdeb4af3c7 P 9dae072a16c44f1a8e4fe720ecff93cc [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:11:34.665930 14618 raft_consensus.cc:686] T da9baf7d3625467bbad07bcdeb4af3c7 P 9dae072a16c44f1a8e4fe720ecff93cc [term 1 LEADER]: Becoming Leader. State: Replica: 9dae072a16c44f1a8e4fe720ecff93cc, State: Running, Role: LEADER
I20260928 02:11:34.666508 14618 consensus_queue.cc:237] T da9baf7d3625467bbad07bcdeb4af3c7 P 9dae072a16c44f1a8e4fe720ecff93cc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dae072a16c44f1a8e4fe720ecff93cc" member_type: VOTER last_known_addr { host: "127.6.235.67" port: 40303 } } peers { permanent_uuid: "9f8e4ca0aeac43aa8ab6f8ab9a4d6fe9" member_type: VOTER last_known_addr { host: "127.6.235.66" port: 42855 } } peers { permanent_uuid: "db7477740b844e9c8e8d14f8724f78d3" member_type: VOTER last_known_addr { host: "127.6.235.65" port: 42113 } }
I20260928 02:11:34.667045 14624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:11:34.666616 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.235.126:34853 (local address 127.6.235.67:47157)
0928 02:11:34.666793 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0928 02:11:34.666901 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.235.126:34853: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260928 02:11:34.667143 14378 tablet.cc:2369] T da9baf7d3625467bbad07bcdeb4af3c7 P 9f8e4ca0aeac43aa8ab6f8ab9a4d6fe9: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 02:11:34.670193 13856 catalog_manager.cc:5495] T da9baf7d3625467bbad07bcdeb4af3c7 P 9dae072a16c44f1a8e4fe720ecff93cc reported cstate change: term changed from 0 to 1, leader changed from <none> to 9dae072a16c44f1a8e4fe720ecff93cc (127.6.235.67). New cstate: current_term: 1 leader_uuid: "9dae072a16c44f1a8e4fe720ecff93cc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9dae072a16c44f1a8e4fe720ecff93cc" member_type: VOTER last_known_addr { host: "127.6.235.67" port: 40303 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9f8e4ca0aeac43aa8ab6f8ab9a4d6fe9" member_type: VOTER last_known_addr { host: "127.6.235.66" port: 42855 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "db7477740b844e9c8e8d14f8724f78d3" member_type: VOTER last_known_addr { host: "127.6.235.65" port: 42113 } health_report { overall_health: UNKNOWN } } }
I20260928 02:11:34.700886 14607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:11:34.695434 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50018 (local address 127.6.235.67:40303)
0928 02:11:34.695634 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0928 02:11:34.695639 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 02:11:34.695658 (+ 19us) server_negotiation.cc:407] Connection header received
0928 02:11:34.695882 (+ 224us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:11:34.695886 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:11:34.695969 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:11:34.696126 (+ 157us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 02:11:34.696498 (+ 372us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:11:34.697252 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:11:34.698520 (+ 1268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:11:34.698928 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:11:34.699094 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:11:34.699517 (+ 423us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 02:11:34.700187 (+ 670us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 02:11:34.700299 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0928 02:11:34.700446 (+ 147us) server_negotiation.cc:299] Negotiation successful
0928 02:11:34.700515 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260928 02:11:34.705667 14330 raft_consensus.cc:1261] T da9baf7d3625467bbad07bcdeb4af3c7 P 9f8e4ca0aeac43aa8ab6f8ab9a4d6fe9 [term 1 FOLLOWER]: Refusing update from remote peer 9dae072a16c44f1a8e4fe720ecff93cc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 02:11:34.706035 14182 raft_consensus.cc:1261] T da9baf7d3625467bbad07bcdeb4af3c7 P db7477740b844e9c8e8d14f8724f78d3 [term 1 FOLLOWER]: Refusing update from remote peer 9dae072a16c44f1a8e4fe720ecff93cc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 02:11:34.706440 14618 consensus_queue.cc:1035] T da9baf7d3625467bbad07bcdeb4af3c7 P 9dae072a16c44f1a8e4fe720ecff93cc [LEADER]: Connected to new peer: Peer: permanent_uuid: "9f8e4ca0aeac43aa8ab6f8ab9a4d6fe9" member_type: VOTER last_known_addr { host: "127.6.235.66" port: 42855 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 02:11:34.706856 14625 consensus_queue.cc:1035] T da9baf7d3625467bbad07bcdeb4af3c7 P 9dae072a16c44f1a8e4fe720ecff93cc [LEADER]: Connected to new peer: Peer: permanent_uuid: "db7477740b844e9c8e8d14f8724f78d3" member_type: VOTER last_known_addr { host: "127.6.235.65" port: 42113 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 02:11:34.709460 14617 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:11:34.708535 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.235.126:34853 (local address 127.6.235.66:46575)
0928 02:11:34.708719 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0928 02:11:34.708852 (+ 133us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.235.126:34853: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
I20260928 02:11:34.710222 14623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:11:34.709724 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.235.126:34853 (local address 127.6.235.65:35875)
0928 02:11:34.709908 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0928 02:11:34.710030 (+ 122us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.235.126:34853: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
I20260928 02:11:34.715830 14623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:11:34.715281 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.235.126:34853 (local address 127.6.235.65:51045)
0928 02:11:34.715468 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0928 02:11:34.715596 (+ 128us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.235.126:34853: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":73}
I20260928 02:11:34.717108 14624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:11:34.716315 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.235.126:34853 (local address 127.6.235.67:38209)
0928 02:11:34.716739 (+ 424us) negotiation.cc:107] Waiting for socket to connect
0928 02:11:34.716858 (+ 119us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.235.126:34853: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":272}
I20260928 02:11:34.728211 14624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:11:34.727622 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.235.126:34853 (local address 127.6.235.67:48695)
0928 02:11:34.727869 (+ 247us) negotiation.cc:107] Waiting for socket to connect
0928 02:11:34.727990 (+ 121us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.235.126:34853: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":124}
I20260928 02:11:34.730621 14617 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:11:34.730122 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.235.126:34853 (local address 127.6.235.66:34749)
0928 02:11:34.730330 (+ 208us) negotiation.cc:107] Waiting for socket to connect
0928 02:11:34.730433 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.235.126:34853: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":84}
I20260928 02:11:34.731372 13857 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60186:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 02:11:34.731645 13857 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60186:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 02:11:34.738538 13857 catalog_manager.cc:5782] T 00000000000000000000000000000000 P bafe67917b9a4b40a60409c753bf4b96: Sending DeleteTablet for 3 replicas of tablet da9baf7d3625467bbad07bcdeb4af3c7
I20260928 02:11:34.740113 14162 tablet_service.cc:1514] Processing DeleteTablet for tablet da9baf7d3625467bbad07bcdeb4af3c7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:11:34 UTC) from {username='oryx', principal='oryx/127.6.235.125@KRBTEST.COM'} at 127.0.0.1:36186
I20260928 02:11:34.740144 14451 tablet_service.cc:1514] Processing DeleteTablet for tablet da9baf7d3625467bbad07bcdeb4af3c7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:11:34 UTC) from {username='oryx', principal='oryx/127.6.235.125@KRBTEST.COM'} at 127.0.0.1:50016
I20260928 02:11:34.740639 14310 tablet_service.cc:1514] Processing DeleteTablet for tablet da9baf7d3625467bbad07bcdeb4af3c7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:11:34 UTC) from {username='oryx', principal='oryx/127.6.235.125@KRBTEST.COM'} at 127.0.0.1:50110
I20260928 02:11:34.740975 14639 tablet_replica.cc:330] T da9baf7d3625467bbad07bcdeb4af3c7 P 9dae072a16c44f1a8e4fe720ecff93cc: stopping tablet replica
I20260928 02:11:34.741215 14639 raft_consensus.cc:2229] T da9baf7d3625467bbad07bcdeb4af3c7 P 9dae072a16c44f1a8e4fe720ecff93cc [term 1 LEADER]: Raft consensus shutting down.
I20260928 02:11:34.741910 14639 raft_consensus.cc:2258] T da9baf7d3625467bbad07bcdeb4af3c7 P 9dae072a16c44f1a8e4fe720ecff93cc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 02:11:34.742616 14638 tablet_replica.cc:330] T da9baf7d3625467bbad07bcdeb4af3c7 P db7477740b844e9c8e8d14f8724f78d3: stopping tablet replica
I20260928 02:11:34.742892 14638 raft_consensus.cc:2229] T da9baf7d3625467bbad07bcdeb4af3c7 P db7477740b844e9c8e8d14f8724f78d3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 02:11:34.743322 14638 raft_consensus.cc:2258] T da9baf7d3625467bbad07bcdeb4af3c7 P db7477740b844e9c8e8d14f8724f78d3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 02:11:34.743381 7085 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskTlaUuc/build/asan/bin/kudu pid 14070
I20260928 02:11:34.744581 14638 ts_tablet_manager.cc:1905] T da9baf7d3625467bbad07bcdeb4af3c7 P db7477740b844e9c8e8d14f8724f78d3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 02:11:34.747692 14639 ts_tablet_manager.cc:1905] T da9baf7d3625467bbad07bcdeb4af3c7 P 9dae072a16c44f1a8e4fe720ecff93cc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 02:11:34.753249 14640 tablet_replica.cc:330] T da9baf7d3625467bbad07bcdeb4af3c7 P 9f8e4ca0aeac43aa8ab6f8ab9a4d6fe9: stopping tablet replica
I20260928 02:11:34.753470 14640 raft_consensus.cc:2229] T da9baf7d3625467bbad07bcdeb4af3c7 P 9f8e4ca0aeac43aa8ab6f8ab9a4d6fe9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 02:11:34.753500 14638 ts_tablet_manager.cc:1918] T da9baf7d3625467bbad07bcdeb4af3c7 P db7477740b844e9c8e8d14f8724f78d3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 02:11:34.753774 14638 log.cc:1196] T da9baf7d3625467bbad07bcdeb4af3c7 P db7477740b844e9c8e8d14f8724f78d3: Deleting WAL directory at /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790561432251112-7085-0/minicluster-data/ts-0/wal/wals/da9baf7d3625467bbad07bcdeb4af3c7
I20260928 02:11:34.753808 14640 raft_consensus.cc:2258] T da9baf7d3625467bbad07bcdeb4af3c7 P 9f8e4ca0aeac43aa8ab6f8ab9a4d6fe9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 02:11:34.754890 14638 ts_tablet_manager.cc:1939] T da9baf7d3625467bbad07bcdeb4af3c7 P db7477740b844e9c8e8d14f8724f78d3: Deleting consensus metadata
I20260928 02:11:34.755009 14640 ts_tablet_manager.cc:1905] T da9baf7d3625467bbad07bcdeb4af3c7 P 9f8e4ca0aeac43aa8ab6f8ab9a4d6fe9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 02:11:34.756311 13842 catalog_manager.cc:4841] TS db7477740b844e9c8e8d14f8724f78d3 (127.6.235.65:42113): tablet da9baf7d3625467bbad07bcdeb4af3c7 (table test-table [id=353a4fbbba284d50a376c6dec5c8e7e6]) successfully deleted
I20260928 02:11:34.762421 14640 ts_tablet_manager.cc:1918] T da9baf7d3625467bbad07bcdeb4af3c7 P 9f8e4ca0aeac43aa8ab6f8ab9a4d6fe9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 02:11:34.762578 14640 log.cc:1196] T da9baf7d3625467bbad07bcdeb4af3c7 P 9f8e4ca0aeac43aa8ab6f8ab9a4d6fe9: Deleting WAL directory at /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790561432251112-7085-0/minicluster-data/ts-1/wal/wals/da9baf7d3625467bbad07bcdeb4af3c7
I20260928 02:11:34.763127 14640 ts_tablet_manager.cc:1939] T da9baf7d3625467bbad07bcdeb4af3c7 P 9f8e4ca0aeac43aa8ab6f8ab9a4d6fe9: Deleting consensus metadata
I20260928 02:11:34.764442 13843 catalog_manager.cc:4841] TS 9f8e4ca0aeac43aa8ab6f8ab9a4d6fe9 (127.6.235.66:42855): tablet da9baf7d3625467bbad07bcdeb4af3c7 (table test-table [id=353a4fbbba284d50a376c6dec5c8e7e6]) successfully deleted
I20260928 02:11:34.767349 14639 ts_tablet_manager.cc:1918] T da9baf7d3625467bbad07bcdeb4af3c7 P 9dae072a16c44f1a8e4fe720ecff93cc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 02:11:34.767556 14639 log.cc:1196] T da9baf7d3625467bbad07bcdeb4af3c7 P 9dae072a16c44f1a8e4fe720ecff93cc: Deleting WAL directory at /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790561432251112-7085-0/minicluster-data/ts-2/wal/wals/da9baf7d3625467bbad07bcdeb4af3c7
I20260928 02:11:34.768160 14639 ts_tablet_manager.cc:1939] T da9baf7d3625467bbad07bcdeb4af3c7 P 9dae072a16c44f1a8e4fe720ecff93cc: Deleting consensus metadata
I20260928 02:11:34.769428 13843 catalog_manager.cc:4841] TS 9dae072a16c44f1a8e4fe720ecff93cc (127.6.235.67:40303): tablet da9baf7d3625467bbad07bcdeb4af3c7 (table test-table [id=353a4fbbba284d50a376c6dec5c8e7e6]) successfully deleted
I20260928 02:11:34.789212 14610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:11:34.744591 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36192 (local address 127.6.235.65:42113)
0928 02:11:34.744977 (+ 386us) server_negotiation.cc:206] Beginning negotiation
0928 02:11:34.744986 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 02:11:34.772953 (+ 27967us) server_negotiation.cc:407] Connection header received
0928 02:11:34.773170 (+ 217us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:11:34.773177 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:11:34.773267 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:11:34.773399 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:11:34.775320 (+ 1921us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:11:34.776457 (+ 1137us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:11:34.778521 (+ 2064us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:11:34.779220 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:11:34.779361 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:11:34.782068 (+ 2707us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:11:34.782087 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:11:34.782111 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:11:34.782169 (+ 58us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:11:34.786158 (+ 3989us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:11:34.786929 (+ 771us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:11:34.786939 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:11:34.786942 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:11:34.787101 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:11:34.787469 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:11:34.787474 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:11:34.787478 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:11:34.787969 (+ 491us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:11:34.788278 (+ 309us) server_negotiation.cc:1035] Waiting for connection context
0928 02:11:34.788477 (+ 199us) server_negotiation.cc:299] Negotiation successful
0928 02:11:34.788758 (+ 281us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":146}
I20260928 02:11:34.790081 14222 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 02:11:34.914182 7085 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTlaUuc/build/asan/bin/kudu with pid 14070
I20260928 02:11:34.938987 7085 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskTlaUuc/build/asan/bin/kudu pid 14239
I20260928 02:11:34.955878 14608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:11:34.939901 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50122 (local address 127.6.235.66:42855)
0928 02:11:34.940082 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0928 02:11:34.940088 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 02:11:34.940180 (+ 92us) server_negotiation.cc:407] Connection header received
0928 02:11:34.940332 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:11:34.940337 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:11:34.940425 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:11:34.940553 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:11:34.941915 (+ 1362us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:11:34.943030 (+ 1115us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:11:34.944573 (+ 1543us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:11:34.945237 (+ 664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:11:34.945404 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:11:34.948126 (+ 2722us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:11:34.948147 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:11:34.948168 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:11:34.948213 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:11:34.952282 (+ 4069us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:11:34.953412 (+ 1130us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:11:34.953423 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:11:34.953427 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:11:34.953601 (+ 174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:11:34.953998 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:11:34.954003 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:11:34.954007 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:11:34.954534 (+ 527us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:11:34.954838 (+ 304us) server_negotiation.cc:1035] Waiting for connection context
0928 02:11:34.955389 (+ 551us) server_negotiation.cc:299] Negotiation successful
0928 02:11:34.955565 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260928 02:11:34.956589 14370 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 02:11:34.987365 14609 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:11:34.986779 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.235.126:34853 (local address 127.0.0.1:43482)
0928 02:11:34.987014 (+ 235us) negotiation.cc:107] Waiting for socket to connect
0928 02:11:34.987148 (+ 134us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.235.126:34853: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":81}
I20260928 02:11:34.988649 14609 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:11:34.988249 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.235.126:34853 (local address 127.0.0.1:43486)
0928 02:11:34.988422 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0928 02:11:34.988516 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.235.126:34853: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
I20260928 02:11:35.053390 7085 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTlaUuc/build/asan/bin/kudu with pid 14239
I20260928 02:11:35.072257 7085 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskTlaUuc/build/asan/bin/kudu pid 14388
I20260928 02:11:35.087576 14607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:11:35.073367 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50032 (local address 127.6.235.67:40303)
0928 02:11:35.073678 (+ 311us) server_negotiation.cc:206] Beginning negotiation
0928 02:11:35.073686 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 02:11:35.073716 (+ 30us) server_negotiation.cc:407] Connection header received
0928 02:11:35.073785 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:11:35.073798 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:11:35.073881 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:11:35.073998 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:11:35.075394 (+ 1396us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:11:35.076188 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:11:35.077580 (+ 1392us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:11:35.078028 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:11:35.078191 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:11:35.080843 (+ 2652us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:11:35.080861 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:11:35.080878 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:11:35.080990 (+ 112us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:11:35.084484 (+ 3494us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:11:35.085297 (+ 813us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:11:35.085311 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:11:35.085314 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:11:35.085494 (+ 180us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:11:35.085895 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:11:35.085900 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:11:35.085903 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:11:35.086296 (+ 393us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:11:35.086619 (+ 323us) server_negotiation.cc:1035] Waiting for connection context
0928 02:11:35.086912 (+ 293us) server_negotiation.cc:299] Negotiation successful
0928 02:11:35.087156 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260928 02:11:35.088402 14511 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 02:11:35.217072 7085 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTlaUuc/build/asan/bin/kudu with pid 14388
I20260928 02:11:35.238626 7085 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskTlaUuc/build/asan/bin/kudu pid 13720
I20260928 02:11:35.239840 13887 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 02:11:35.362227 7085 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTlaUuc/build/asan/bin/kudu with pid 13720
W20260928 02:11:35.384414 14029 connection.cc:535] server connection from 127.0.0.1:41066 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260928 02:11:35.384828 7085 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskTlaUuc/build/asan/bin/kudu pid 13821
I20260928 02:11:35.385952 14076 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 02:11:35.502787 7085 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTlaUuc/build/asan/bin/kudu with pid 13821
2026-09-28T02:11:35Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipalMultipleMaster (9979 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790561432251112-7085-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 02:11:35 dist-test-slave-86gv krb5kdc[14649](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 02:11:35 dist-test-slave-86gv krb5kdc[14649](info): set up 2 sockets
Sep 28 02:11:35 dist-test-slave-86gv krb5kdc[14649](info): commencing operation
krb5kdc: starting...
W20260928 02:11:36.702436 7085 mini_kdc.cc:121] Time spent starting KDC: real 1.151s user 0.000s sys 0.023s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790561432251112-7085-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790561432251112-7085-0/krb5kdc/test-user.keytab.
Sep 28 02:11:36 dist-test-slave-86gv krb5kdc[14649](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561496, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T02:11:36Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T02:11:36Z Disabled control of system clock
WARNING: no policy specified for oryx/127.6.235.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.6.235.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.6.235.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790561432251112-7085-0/krb5kdc/oryx_127.6.235.126.keytab.
Entry for principal oryx/127.6.235.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790561432251112-7085-0/krb5kdc/oryx_127.6.235.126.keytab.
I20260928 02:11:36.941555 7085 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTlaUuc/build/asan/bin/kudu
/tmp/dist-test-taskTlaUuc/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790561432251112-7085-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790561432251112-7085-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790561432251112-7085-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790561432251112-7085-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.235.126:32895
--webserver_interface=127.6.235.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790561432251112-7085-0/krb5kdc/oryx_127.6.235.126.keytab
--principal=oryx/127.6.235.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.235.84:35667
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.235.126:32895
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790561432251112-7085-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790561432251112-7085-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790561432251112-7085-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:11:37.281121 14662 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:11:37.281445 14662 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:11:37.281613 14662 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:11:37.289139 14662 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 02:11:37.289261 14662 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260928 02:11:37.289371 14662 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:11:37.289497 14662 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:11:37.289598 14662 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 02:11:37.289696 14662 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 02:11:37.299998 14662 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.235.84:35667
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790561432251112-7085-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790561432251112-7085-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.235.126:32895
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790561432251112-7085-0/krb5kdc/oryx_127.6.235.126.keytab
--principal=oryx/127.6.235.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.235.126:32895
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790561432251112-7085-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.235.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790561432251112-7085-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 00:51:59 UTC on ccycloud.quasar-axgkwn.root.comops.site
build id 4676464
ASAN enabled
I20260928 02:11:37.301477 14662 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:11:37.302983 14662 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:11:37.316071 14667 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:11:37.316063 14670 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:11:37.316063 14668 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:11:37.316920 14662 server_base.cc:1034] running on GCE node
I20260928 02:11:37.317768 14662 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:11:37.319306 14662 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:11:37.320526 14662 hybrid_clock.cc:648] HybridClock initialized: now 1790561497320440 us; error 65 us; skew 500 ppm
I20260928 02:11:37.320863 14662 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 02:11:37 dist-test-slave-86gv krb5kdc[14649](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561497, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.235.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:11:37.324194 14662 init.cc:377] Logged in from keytab as oryx/127.6.235.126@KRBTEST.COM (short username oryx)
I20260928 02:11:37.325891 14662 webserver.cc:460] Webserver started at http://127.6.235.126:37685/ using document root <none> and password file <none>
I20260928 02:11:37.326651 14662 fs_manager.cc:359] Metadata directory not provided
I20260928 02:11:37.326743 14662 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:11:37.327085 14662 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:11:37.330282 14662 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790561432251112-7085-0/minicluster-data/master-0/data/instance:
uuid: "03972cb2d8744214a6a4570fb5531fe0"
format_stamp: "Formatted at 2026-09-28 02:11:37 on dist-test-slave-86gv"
server_key: "9a846ff5b44736b96bb7b6689724f4ff"
server_key_iv: "655681e059ebe4d897cd44df5d85cc9b"
server_key_version: "encryptionkey@0"
I20260928 02:11:37.331099 14662 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790561432251112-7085-0/minicluster-data/master-0/wal/instance:
uuid: "03972cb2d8744214a6a4570fb5531fe0"
format_stamp: "Formatted at 2026-09-28 02:11:37 on dist-test-slave-86gv"
server_key: "9a846ff5b44736b96bb7b6689724f4ff"
server_key_iv: "655681e059ebe4d897cd44df5d85cc9b"
server_key_version: "encryptionkey@0"
I20260928 02:11:37.336443 14662 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260928 02:11:37.340173 14677 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:11:37.341706 14662 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260928 02:11:37.341912 14662 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790561432251112-7085-0/minicluster-data/master-0/data,/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790561432251112-7085-0/minicluster-data/master-0/wal
uuid: "03972cb2d8744214a6a4570fb5531fe0"
format_stamp: "Formatted at 2026-09-28 02:11:37 on dist-test-slave-86gv"
server_key: "9a846ff5b44736b96bb7b6689724f4ff"
server_key_iv: "655681e059ebe4d897cd44df5d85cc9b"
server_key_version: "encryptionkey@0"
I20260928 02:11:37.342182 14662 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790561432251112-7085-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790561432251112-7085-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790561432251112-7085-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:11:37.375489 14662 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:11:37.380424 14662 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:11:37.380750 14662 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:11:37.403427 14662 rpc_server.cc:307] RPC server started. Bound to: 127.6.235.126:32895
I20260928 02:11:37.403447 14738 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.235.126:32895 every 8 connection(s)
I20260928 02:11:37.404865 14662 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790561432251112-7085-0/minicluster-data/master-0/data/info.pb
I20260928 02:11:37.405977 7085 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTlaUuc/build/asan/bin/kudu as pid 14662
I20260928 02:11:37.406225 7085 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790561432251112-7085-0/minicluster-data/master-0/wal/instance
I20260928 02:11:37.407022 7085 external_mini_cluster.cc:1428] Setting key b0ae45df9e6d1c93419d9c42bd0eded5
I20260928 02:11:37.410051 14739 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 28 02:11:37 dist-test-slave-86gv krb5kdc[14649](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561496, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.6.235.126@KRBTEST.COM
I20260928 02:11:37.423753 14739 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 03972cb2d8744214a6a4570fb5531fe0: Bootstrap starting.
I20260928 02:11:37.427130 14739 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 03972cb2d8744214a6a4570fb5531fe0: Neither blocks nor log segments found. Creating new log.
I20260928 02:11:37.428251 14739 log.cc:824] T 00000000000000000000000000000000 P 03972cb2d8744214a6a4570fb5531fe0: Log is configured to *not* fsync() on all Append() calls
I20260928 02:11:37.431180 14743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:11:37.409310 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55426 (local address 127.6.235.126:32895)
0928 02:11:37.410172 (+ 862us) server_negotiation.cc:206] Beginning negotiation
0928 02:11:37.410187 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0928 02:11:37.410242 (+ 55us) server_negotiation.cc:407] Connection header received
0928 02:11:37.411848 (+ 1606us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:11:37.411871 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:11:37.412267 (+ 396us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:11:37.412610 (+ 343us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:11:37.414484 (+ 1874us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:11:37.415892 (+ 1408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:11:37.417708 (+ 1816us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:11:37.418392 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:11:37.418530 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:11:37.423324 (+ 4794us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:11:37.423332 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:11:37.423353 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:11:37.423398 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:11:37.426483 (+ 3085us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:11:37.427259 (+ 776us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:11:37.427266 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:11:37.427273 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:11:37.427485 (+ 212us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:11:37.427886 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:11:37.427891 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:11:37.427895 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:11:37.428503 (+ 608us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:11:37.428803 (+ 300us) server_negotiation.cc:1035] Waiting for connection context
0928 02:11:37.429321 (+ 518us) server_negotiation.cc:299] Negotiation successful
0928 02:11:37.429818 (+ 497us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":607,"thread_start_us":398,"threads_started":1}
I20260928 02:11:37.431792 14739 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 03972cb2d8744214a6a4570fb5531fe0: No bootstrap required, opened a new log
I20260928 02:11:37.440096 14739 raft_consensus.cc:348] T 00000000000000000000000000000000 P 03972cb2d8744214a6a4570fb5531fe0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "03972cb2d8744214a6a4570fb5531fe0" member_type: VOTER last_known_addr { host: "127.6.235.126" port: 32895 } }
I20260928 02:11:37.440474 14739 raft_consensus.cc:374] T 00000000000000000000000000000000 P 03972cb2d8744214a6a4570fb5531fe0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:11:37.440549 14739 raft_consensus.cc:729] T 00000000000000000000000000000000 P 03972cb2d8744214a6a4570fb5531fe0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 03972cb2d8744214a6a4570fb5531fe0, State: Initialized, Role: FOLLOWER
I20260928 02:11:37.441157 14739 consensus_queue.cc:260] T 00000000000000000000000000000000 P 03972cb2d8744214a6a4570fb5531fe0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "03972cb2d8744214a6a4570fb5531fe0" member_type: VOTER last_known_addr { host: "127.6.235.126" port: 32895 } }
I20260928 02:11:37.441368 14739 raft_consensus.cc:388] T 00000000000000000000000000000000 P 03972cb2d8744214a6a4570fb5531fe0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 02:11:37.441466 14739 raft_consensus.cc:482] T 00000000000000000000000000000000 P 03972cb2d8744214a6a4570fb5531fe0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 02:11:37.441605 14739 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 03972cb2d8744214a6a4570fb5531fe0 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:11:37.443815 14739 raft_consensus.cc:504] T 00000000000000000000000000000000 P 03972cb2d8744214a6a4570fb5531fe0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "03972cb2d8744214a6a4570fb5531fe0" member_type: VOTER last_known_addr { host: "127.6.235.126" port: 32895 } }
I20260928 02:11:37.444439 14739 leader_election.cc:302] T 00000000000000000000000000000000 P 03972cb2d8744214a6a4570fb5531fe0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 03972cb2d8744214a6a4570fb5531fe0; no voters:
I20260928 02:11:37.445112 14739 leader_election.cc:288] T 00000000000000000000000000000000 P 03972cb2d8744214a6a4570fb5531fe0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 02:11:37.445242 14745 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 03972cb2d8744214a6a4570fb5531fe0 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:11:37.446084 14745 raft_consensus.cc:686] T 00000000000000000000000000000000 P 03972cb2d8744214a6a4570fb5531fe0 [term 1 LEADER]: Becoming Leader. State: Replica: 03972cb2d8744214a6a4570fb5531fe0, State: Running, Role: LEADER
I20260928 02:11:37.446724 14745 consensus_queue.cc:237] T 00000000000000000000000000000000 P 03972cb2d8744214a6a4570fb5531fe0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "03972cb2d8744214a6a4570fb5531fe0" member_type: VOTER last_known_addr { host: "127.6.235.126" port: 32895 } }
I20260928 02:11:37.447463 14739 sys_catalog.cc:564] T 00000000000000000000000000000000 P 03972cb2d8744214a6a4570fb5531fe0 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 02:11:37.450407 14747 sys_catalog.cc:455] T 00000000000000000000000000000000 P 03972cb2d8744214a6a4570fb5531fe0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 03972cb2d8744214a6a4570fb5531fe0. Latest consensus state: current_term: 1 leader_uuid: "03972cb2d8744214a6a4570fb5531fe0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "03972cb2d8744214a6a4570fb5531fe0" member_type: VOTER last_known_addr { host: "127.6.235.126" port: 32895 } } }
I20260928 02:11:37.451018 14747 sys_catalog.cc:458] T 00000000000000000000000000000000 P 03972cb2d8744214a6a4570fb5531fe0 [sys.catalog]: This master's current role is: LEADER
I20260928 02:11:37.450747 14746 sys_catalog.cc:455] T 00000000000000000000000000000000 P 03972cb2d8744214a6a4570fb5531fe0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "03972cb2d8744214a6a4570fb5531fe0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "03972cb2d8744214a6a4570fb5531fe0" member_type: VOTER last_known_addr { host: "127.6.235.126" port: 32895 } } }
I20260928 02:11:37.453943 14746 sys_catalog.cc:458] T 00000000000000000000000000000000 P 03972cb2d8744214a6a4570fb5531fe0 [sys.catalog]: This master's current role is: LEADER
I20260928 02:11:37.454761 14754 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 02:11:37.460021 14754 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 02:11:37.469775 14754 catalog_manager.cc:1347] Generated new cluster ID: 78597842d1be4cfda933e3c94692593b
I20260928 02:11:37.469920 14754 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 02:11:37.479439 14754 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 02:11:37.480800 14754 catalog_manager.cc:1504] Loading token signing keys...
I20260928 02:11:37.493424 14754 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 03972cb2d8744214a6a4570fb5531fe0: Generated new TSK 0
I20260928 02:11:37.494375 14754 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.6.235.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.6.235.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.6.235.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790561432251112-7085-0/krb5kdc/oryx_127.6.235.65.keytab.
Entry for principal oryx/127.6.235.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790561432251112-7085-0/krb5kdc/oryx_127.6.235.65.keytab.
I20260928 02:11:37.558917 7085 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTlaUuc/build/asan/bin/kudu
/tmp/dist-test-taskTlaUuc/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790561432251112-7085-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790561432251112-7085-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790561432251112-7085-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790561432251112-7085-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.235.65:0
--local_ip_for_outbound_sockets=127.6.235.65
--webserver_interface=127.6.235.65
--webserver_port=0
--tserver_master_addrs=127.6.235.126:32895
--keytab_file=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790561432251112-7085-0/krb5kdc/oryx_127.6.235.65.keytab
--principal=oryx/127.6.235.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.235.84:35667
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790561432251112-7085-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790561432251112-7085-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790561432251112-7085-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:11:37.918735 14766 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:11:37.919092 14766 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:11:37.919288 14766 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:11:37.926798 14766 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:11:37.926964 14766 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:11:37.927140 14766 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260928 02:11:37.927289 14766 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.235.65
I20260928 02:11:37.937960 14766 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.235.84:35667
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790561432251112-7085-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790561432251112-7085-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790561432251112-7085-0/krb5kdc/oryx_127.6.235.65.keytab
--principal=oryx/127.6.235.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.235.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790561432251112-7085-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.6.235.65
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.6.235.126:32895
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.235.65
--log_dir=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790561432251112-7085-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 00:51:59 UTC on ccycloud.quasar-axgkwn.root.comops.site
build id 4676464
ASAN enabled
I20260928 02:11:37.939411 14766 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:11:37.940826 14766 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:11:37.955516 14772 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:11:37.955673 14771 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:11:37.955446 14774 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:11:37.956302 14766 server_base.cc:1034] running on GCE node
I20260928 02:11:37.957127 14766 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:11:37.958210 14766 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:11:37.959446 14766 hybrid_clock.cc:648] HybridClock initialized: now 1790561497959375 us; error 41 us; skew 500 ppm
I20260928 02:11:37.959896 14766 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 02:11:37 dist-test-slave-86gv krb5kdc[14649](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561497, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.235.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:11:37.969976 14766 init.cc:377] Logged in from keytab as oryx/127.6.235.65@KRBTEST.COM (short username oryx)
I20260928 02:11:37.971791 14766 webserver.cc:460] Webserver started at http://127.6.235.65:36177/ using document root <none> and password file <none>
I20260928 02:11:37.972693 14766 fs_manager.cc:359] Metadata directory not provided
I20260928 02:11:37.972849 14766 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:11:37.973366 14766 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:11:37.976588 14766 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790561432251112-7085-0/minicluster-data/ts-0/data/instance:
uuid: "6e7411d0f5db463799e9b2db6669ea48"
format_stamp: "Formatted at 2026-09-28 02:11:37 on dist-test-slave-86gv"
server_key: "43dc5ef34d1232cb22db951244df9f39"
server_key_iv: "276987eb903082a2273aa2eb40cf910a"
server_key_version: "encryptionkey@0"
I20260928 02:11:37.977486 14766 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790561432251112-7085-0/minicluster-data/ts-0/wal/instance:
uuid: "6e7411d0f5db463799e9b2db6669ea48"
format_stamp: "Formatted at 2026-09-28 02:11:37 on dist-test-slave-86gv"
server_key: "43dc5ef34d1232cb22db951244df9f39"
server_key_iv: "276987eb903082a2273aa2eb40cf910a"
server_key_version: "encryptionkey@0"
I20260928 02:11:37.982978 14766 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260928 02:11:37.986863 14781 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:11:37.988430 14766 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260928 02:11:37.988608 14766 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790561432251112-7085-0/minicluster-data/ts-0/data,/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790561432251112-7085-0/minicluster-data/ts-0/wal
uuid: "6e7411d0f5db463799e9b2db6669ea48"
format_stamp: "Formatted at 2026-09-28 02:11:37 on dist-test-slave-86gv"
server_key: "43dc5ef34d1232cb22db951244df9f39"
server_key_iv: "276987eb903082a2273aa2eb40cf910a"
server_key_version: "encryptionkey@0"
I20260928 02:11:37.988839 14766 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790561432251112-7085-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790561432251112-7085-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790561432251112-7085-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:11:38.016420 14766 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:11:38.020830 14766 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:11:38.021190 14766 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:11:38.024263 14766 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:11:38.024348 14766 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:11:38.024537 14766 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:11:38.024596 14766 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 28 02:11:38 dist-test-slave-86gv krb5kdc[14649](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561497, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.235.65@KRBTEST.COM for oryx/127.6.235.126@KRBTEST.COM
I20260928 02:11:38.052562 14790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:11:38.025014 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.235.65:59673 (local address 127.6.235.126:32895)
0928 02:11:38.026764 (+ 1750us) server_negotiation.cc:206] Beginning negotiation
0928 02:11:38.026769 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 02:11:38.027174 (+ 405us) server_negotiation.cc:407] Connection header received
0928 02:11:38.029558 (+ 2384us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:11:38.029564 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:11:38.029652 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:11:38.029800 (+ 148us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:11:38.031360 (+ 1560us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:11:38.032657 (+ 1297us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:11:38.035558 (+ 2901us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:11:38.036052 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:11:38.036204 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:11:38.042653 (+ 6449us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:11:38.042665 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:11:38.042668 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:11:38.042705 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:11:38.046485 (+ 3780us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:11:38.048100 (+ 1615us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:11:38.048110 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:11:38.048114 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:11:38.048299 (+ 185us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:11:38.049017 (+ 718us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:11:38.049025 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:11:38.049031 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:11:38.049367 (+ 336us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:11:38.049531 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0928 02:11:38.051979 (+ 2448us) server_negotiation.cc:299] Negotiation successful
0928 02:11:38.052225 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1629,"thread_start_us":1525,"threads_started":1}
I20260928 02:11:38.053129 14791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:11:38.025977 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.235.126:32895 (local address 127.6.235.65:59673)
0928 02:11:38.027005 (+ 1028us) negotiation.cc:107] Waiting for socket to connect
0928 02:11:38.027051 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0928 02:11:38.029190 (+ 2139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:11:38.030038 (+ 848us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:11:38.030046 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:11:38.030561 (+ 515us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:11:38.031179 (+ 618us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:11:38.031197 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:11:38.032828 (+ 1631us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:11:38.032832 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:11:38.035383 (+ 2551us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:11:38.035396 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:11:38.036368 (+ 972us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:11:38.036378 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:11:38.036841 (+ 463us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:11:38.037830 (+ 989us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:11:38.037878 (+ 48us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:11:38.042286 (+ 4408us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:11:38.046722 (+ 4436us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:11:38.046737 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:11:38.046771 (+ 34us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:11:38.047898 (+ 1127us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:11:38.048443 (+ 545us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:11:38.048451 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:11:38.048458 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:11:38.048830 (+ 372us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:11:38.049511 (+ 681us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:11:38.049523 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:11:38.050070 (+ 547us) client_negotiation.cc:769] Sending connection context
0928 02:11:38.050538 (+ 468us) client_negotiation.cc:240] Negotiation successful
0928 02:11:38.051188 (+ 650us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":740,"thread_start_us":485,"threads_started":1}
I20260928 02:11:38.069689 14766 rpc_server.cc:307] RPC server started. Bound to: 127.6.235.65:42571
I20260928 02:11:38.069739 14902 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.235.65:42571 every 8 connection(s)
I20260928 02:11:38.071051 14766 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790561432251112-7085-0/minicluster-data/ts-0/data/info.pb
I20260928 02:11:38.073460 14903 heartbeater.cc:347] Connected to a master server at 127.6.235.126:32895
I20260928 02:11:38.073786 14903 heartbeater.cc:464] Registering TS with master...
I20260928 02:11:38.074662 14903 heartbeater.cc:511] Master 127.6.235.126:32895 requested a full tablet report, sending...
I20260928 02:11:38.076231 14880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:11:38.060458 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.235.126:32895 (local address 127.6.235.65:36437)
0928 02:11:38.061321 (+ 863us) negotiation.cc:107] Waiting for socket to connect
0928 02:11:38.061349 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0928 02:11:38.061600 (+ 251us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:11:38.061999 (+ 399us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:11:38.062006 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:11:38.062922 (+ 916us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:11:38.063561 (+ 639us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:11:38.063589 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:11:38.064968 (+ 1379us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:11:38.064975 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:11:38.066776 (+ 1801us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:11:38.066792 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:11:38.067715 (+ 923us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:11:38.067721 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:11:38.067887 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:11:38.068653 (+ 766us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:11:38.068685 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:11:38.069644 (+ 959us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:11:38.073297 (+ 3653us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:11:38.073305 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:11:38.073311 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:11:38.073924 (+ 613us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:11:38.074322 (+ 398us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:11:38.074328 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:11:38.074332 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:11:38.074480 (+ 148us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:11:38.075095 (+ 615us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:11:38.075100 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:11:38.075365 (+ 265us) client_negotiation.cc:769] Sending connection context
0928 02:11:38.075504 (+ 139us) client_negotiation.cc:240] Negotiation successful
0928 02:11:38.075839 (+ 335us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":659,"thread_start_us":477,"threads_started":1}
I20260928 02:11:38.076267 14790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:11:38.060467 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.235.65:36437 (local address 127.6.235.126:32895)
0928 02:11:38.060699 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0928 02:11:38.060708 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 02:11:38.061454 (+ 746us) server_negotiation.cc:407] Connection header received
0928 02:11:38.061769 (+ 315us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:11:38.061774 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:11:38.061838 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:11:38.061922 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:11:38.063775 (+ 1853us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:11:38.064740 (+ 965us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:11:38.066940 (+ 2200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:11:38.067494 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:11:38.067687 (+ 193us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:11:38.069847 (+ 2160us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:11:38.069854 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:11:38.069859 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:11:38.069903 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:11:38.073142 (+ 3239us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:11:38.074052 (+ 910us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:11:38.074056 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:11:38.074058 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:11:38.074198 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:11:38.074642 (+ 444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:11:38.074645 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:11:38.074647 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:11:38.074974 (+ 327us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:11:38.075103 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0928 02:11:38.075691 (+ 588us) server_negotiation.cc:299] Negotiation successful
0928 02:11:38.075878 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260928 02:11:38.077312 14694 ts_manager.cc:194] Registered new tserver with Master: 6e7411d0f5db463799e9b2db6669ea48 (127.6.235.65:42571)
I20260928 02:11:38.077754 7085 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTlaUuc/build/asan/bin/kudu as pid 14766
I20260928 02:11:38.077973 7085 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790561432251112-7085-0/minicluster-data/ts-0/wal/instance
I20260928 02:11:38.078740 7085 external_mini_cluster.cc:1428] Setting key 69f674d9673818e108f1bf386ef5b513
I20260928 02:11:38.079936 14694 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.6.235.65@KRBTEST.COM'} at 127.6.235.65:59673
WARNING: no policy specified for oryx/127.6.235.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.6.235.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.6.235.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790561432251112-7085-0/krb5kdc/oryx_127.6.235.66.keytab.
Entry for principal oryx/127.6.235.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790561432251112-7085-0/krb5kdc/oryx_127.6.235.66.keytab.
I20260928 02:11:38.129210 7085 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTlaUuc/build/asan/bin/kudu
/tmp/dist-test-taskTlaUuc/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790561432251112-7085-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790561432251112-7085-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790561432251112-7085-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790561432251112-7085-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.235.66:0
--local_ip_for_outbound_sockets=127.6.235.66
--webserver_interface=127.6.235.66
--webserver_port=0
--tserver_master_addrs=127.6.235.126:32895
--keytab_file=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790561432251112-7085-0/krb5kdc/oryx_127.6.235.66.keytab
--principal=oryx/127.6.235.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.235.84:35667
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790561432251112-7085-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790561432251112-7085-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790561432251112-7085-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:11:38.476441 14907 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:11:38.476747 14907 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:11:38.476982 14907 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:11:38.484449 14907 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:11:38.484617 14907 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:11:38.484781 14907 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260928 02:11:38.484951 14907 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.235.66
I20260928 02:11:38.495178 14907 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.235.84:35667
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790561432251112-7085-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790561432251112-7085-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790561432251112-7085-0/krb5kdc/oryx_127.6.235.66.keytab
--principal=oryx/127.6.235.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.235.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790561432251112-7085-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.6.235.66
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.6.235.126:32895
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.235.66
--log_dir=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790561432251112-7085-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 00:51:59 UTC on ccycloud.quasar-axgkwn.root.comops.site
build id 4676464
ASAN enabled
I20260928 02:11:38.496573 14907 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:11:38.497984 14907 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:11:38.512004 14912 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:11:38.512009 14913 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:11:38.512935 14915 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:11:38.513870 14907 server_base.cc:1034] running on GCE node
I20260928 02:11:38.514695 14907 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:11:38.515630 14907 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:11:38.517093 14907 hybrid_clock.cc:648] HybridClock initialized: now 1790561498516851 us; error 39 us; skew 500 ppm
I20260928 02:11:38.517443 14907 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 02:11:38 dist-test-slave-86gv krb5kdc[14649](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561498, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.235.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:11:38.520987 14907 init.cc:377] Logged in from keytab as oryx/127.6.235.66@KRBTEST.COM (short username oryx)
I20260928 02:11:38.522491 14907 webserver.cc:460] Webserver started at http://127.6.235.66:38283/ using document root <none> and password file <none>
I20260928 02:11:38.523295 14907 fs_manager.cc:359] Metadata directory not provided
I20260928 02:11:38.523383 14907 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:11:38.523648 14907 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:11:38.526629 14907 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790561432251112-7085-0/minicluster-data/ts-1/data/instance:
uuid: "476becc74e6c42449c408fc8bc3380b1"
format_stamp: "Formatted at 2026-09-28 02:11:38 on dist-test-slave-86gv"
server_key: "7b88d512cb0a6c761ca6be02df1b5728"
server_key_iv: "63531622676d8d11e1de4f106527c81e"
server_key_version: "encryptionkey@0"
I20260928 02:11:38.527364 14907 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790561432251112-7085-0/minicluster-data/ts-1/wal/instance:
uuid: "476becc74e6c42449c408fc8bc3380b1"
format_stamp: "Formatted at 2026-09-28 02:11:38 on dist-test-slave-86gv"
server_key: "7b88d512cb0a6c761ca6be02df1b5728"
server_key_iv: "63531622676d8d11e1de4f106527c81e"
server_key_version: "encryptionkey@0"
I20260928 02:11:38.533031 14907 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260928 02:11:38.536641 14922 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:11:38.538106 14907 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 02:11:38.538280 14907 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790561432251112-7085-0/minicluster-data/ts-1/data,/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790561432251112-7085-0/minicluster-data/ts-1/wal
uuid: "476becc74e6c42449c408fc8bc3380b1"
format_stamp: "Formatted at 2026-09-28 02:11:38 on dist-test-slave-86gv"
server_key: "7b88d512cb0a6c761ca6be02df1b5728"
server_key_iv: "63531622676d8d11e1de4f106527c81e"
server_key_version: "encryptionkey@0"
I20260928 02:11:38.538498 14907 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790561432251112-7085-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790561432251112-7085-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790561432251112-7085-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:11:38.555117 14907 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:11:38.559826 14907 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:11:38.560120 14907 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:11:38.563112 14907 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:11:38.563226 14907 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:11:38.563390 14907 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:11:38.563470 14907 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 28 02:11:38 dist-test-slave-86gv krb5kdc[14649](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561498, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.235.66@KRBTEST.COM for oryx/127.6.235.126@KRBTEST.COM
I20260928 02:11:38.589041 14790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:11:38.563939 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.235.66:34167 (local address 127.6.235.126:32895)
0928 02:11:38.564127 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0928 02:11:38.564133 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 02:11:38.565726 (+ 1593us) server_negotiation.cc:407] Connection header received
0928 02:11:38.569135 (+ 3409us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:11:38.569143 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:11:38.569243 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:11:38.569385 (+ 142us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:11:38.571905 (+ 2520us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:11:38.573265 (+ 1360us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:11:38.575609 (+ 2344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:11:38.576230 (+ 621us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:11:38.576376 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:11:38.582113 (+ 5737us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:11:38.582120 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:11:38.582125 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:11:38.582160 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:11:38.585388 (+ 3228us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:11:38.586513 (+ 1125us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:11:38.586517 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:11:38.586519 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:11:38.586686 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:11:38.587243 (+ 557us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:11:38.587251 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:11:38.587256 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:11:38.587541 (+ 285us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:11:38.587716 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0928 02:11:38.588445 (+ 729us) server_negotiation.cc:299] Negotiation successful
0928 02:11:38.588662 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260928 02:11:38.590698 14931 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:11:38.564529 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.235.126:32895 (local address 127.6.235.66:34167)
0928 02:11:38.565522 (+ 993us) negotiation.cc:107] Waiting for socket to connect
0928 02:11:38.565575 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0928 02:11:38.568676 (+ 3101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:11:38.569686 (+ 1010us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:11:38.569702 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:11:38.570677 (+ 975us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:11:38.571660 (+ 983us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:11:38.571709 (+ 49us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:11:38.573457 (+ 1748us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:11:38.573465 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:11:38.575387 (+ 1922us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:11:38.575408 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:11:38.576500 (+ 1092us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:11:38.576513 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:11:38.576766 (+ 253us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:11:38.577799 (+ 1033us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:11:38.577821 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:11:38.581822 (+ 4001us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:11:38.585611 (+ 3789us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:11:38.585635 (+ 24us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:11:38.585656 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:11:38.586335 (+ 679us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:11:38.586843 (+ 508us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:11:38.586853 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:11:38.586856 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:11:38.587112 (+ 256us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:11:38.587691 (+ 579us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:11:38.587700 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:11:38.588078 (+ 378us) client_negotiation.cc:769] Sending connection context
0928 02:11:38.588365 (+ 287us) client_negotiation.cc:240] Negotiation successful
0928 02:11:38.588952 (+ 587us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":678,"thread_start_us":465,"threads_started":1}
I20260928 02:11:38.611096 14790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:11:38.596611 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.235.66:37027 (local address 127.6.235.126:32895)
0928 02:11:38.596815 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0928 02:11:38.596824 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 02:11:38.597593 (+ 769us) server_negotiation.cc:407] Connection header received
0928 02:11:38.597929 (+ 336us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:11:38.597933 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:11:38.597992 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:11:38.598099 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:11:38.599019 (+ 920us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:11:38.599732 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:11:38.601251 (+ 1519us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:11:38.601836 (+ 585us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:11:38.602006 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:11:38.604782 (+ 2776us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:11:38.604787 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:11:38.604791 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:11:38.604821 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:11:38.607485 (+ 2664us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:11:38.608497 (+ 1012us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:11:38.608505 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:11:38.608509 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:11:38.608668 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:11:38.609179 (+ 511us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:11:38.609186 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:11:38.609195 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:11:38.609779 (+ 584us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:11:38.609962 (+ 183us) server_negotiation.cc:1035] Waiting for connection context
0928 02:11:38.610453 (+ 491us) server_negotiation.cc:299] Negotiation successful
0928 02:11:38.610691 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260928 02:11:38.611209 15001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:11:38.596613 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.235.126:32895 (local address 127.6.235.66:37027)
0928 02:11:38.597451 (+ 838us) negotiation.cc:107] Waiting for socket to connect
0928 02:11:38.597501 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0928 02:11:38.597727 (+ 226us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:11:38.598133 (+ 406us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:11:38.598138 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:11:38.598604 (+ 466us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:11:38.598877 (+ 273us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:11:38.598885 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:11:38.599871 (+ 986us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:11:38.599876 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:11:38.601035 (+ 1159us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:11:38.601048 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:11:38.602094 (+ 1046us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:11:38.602102 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:11:38.602375 (+ 273us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:11:38.603298 (+ 923us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:11:38.603318 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:11:38.604561 (+ 1243us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:11:38.607633 (+ 3072us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:11:38.607638 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:11:38.607642 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:11:38.608309 (+ 667us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:11:38.608798 (+ 489us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:11:38.608805 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:11:38.608809 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:11:38.609040 (+ 231us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:11:38.609944 (+ 904us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:11:38.609950 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:11:38.610164 (+ 214us) client_negotiation.cc:769] Sending connection context
0928 02:11:38.610318 (+ 154us) client_negotiation.cc:240] Negotiation successful
0928 02:11:38.610704 (+ 386us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":672,"thread_start_us":500,"threads_started":1}
I20260928 02:11:38.621193 14907 rpc_server.cc:307] RPC server started. Bound to: 127.6.235.66:41335
I20260928 02:11:38.621227 15045 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.235.66:41335 every 8 connection(s)
I20260928 02:11:38.622467 14907 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790561432251112-7085-0/minicluster-data/ts-1/data/info.pb
I20260928 02:11:38.624183 7085 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTlaUuc/build/asan/bin/kudu as pid 14907
I20260928 02:11:38.624395 7085 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790561432251112-7085-0/minicluster-data/ts-1/wal/instance
I20260928 02:11:38.624873 15046 heartbeater.cc:347] Connected to a master server at 127.6.235.126:32895
I20260928 02:11:38.624997 7085 external_mini_cluster.cc:1428] Setting key 51a2ff38e120465c368c9428f5317d02
I20260928 02:11:38.625195 15046 heartbeater.cc:464] Registering TS with master...
I20260928 02:11:38.626022 15046 heartbeater.cc:511] Master 127.6.235.126:32895 requested a full tablet report, sending...
I20260928 02:11:38.627527 14694 ts_manager.cc:194] Registered new tserver with Master: 476becc74e6c42449c408fc8bc3380b1 (127.6.235.66:41335)
I20260928 02:11:38.628381 14694 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.6.235.66@KRBTEST.COM'} at 127.6.235.66:34167
WARNING: no policy specified for oryx/127.6.235.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.6.235.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.6.235.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790561432251112-7085-0/krb5kdc/oryx_127.6.235.67.keytab.
Entry for principal oryx/127.6.235.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790561432251112-7085-0/krb5kdc/oryx_127.6.235.67.keytab.
I20260928 02:11:38.677796 7085 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTlaUuc/build/asan/bin/kudu
/tmp/dist-test-taskTlaUuc/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790561432251112-7085-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790561432251112-7085-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790561432251112-7085-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790561432251112-7085-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.235.67:0
--local_ip_for_outbound_sockets=127.6.235.67
--webserver_interface=127.6.235.67
--webserver_port=0
--tserver_master_addrs=127.6.235.126:32895
--keytab_file=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790561432251112-7085-0/krb5kdc/oryx_127.6.235.67.keytab
--principal=oryx/127.6.235.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.235.84:35667
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790561432251112-7085-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790561432251112-7085-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790561432251112-7085-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:11:39.019624 15050 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:11:39.020037 15050 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:11:39.020162 15050 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:11:39.027704 15050 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:11:39.027829 15050 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:11:39.027921 15050 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260928 02:11:39.028010 15050 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.235.67
I20260928 02:11:39.037832 15050 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.235.84:35667
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790561432251112-7085-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790561432251112-7085-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790561432251112-7085-0/krb5kdc/oryx_127.6.235.67.keytab
--principal=oryx/127.6.235.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.235.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790561432251112-7085-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.6.235.67
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.6.235.126:32895
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.235.67
--log_dir=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790561432251112-7085-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 00:51:59 UTC on ccycloud.quasar-axgkwn.root.comops.site
build id 4676464
ASAN enabled
I20260928 02:11:39.039144 15050 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:11:39.040551 15050 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:11:39.055388 15056 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:11:39.055513 15058 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:11:39.055433 15055 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:11:39.056507 15050 server_base.cc:1034] running on GCE node
I20260928 02:11:39.057300 15050 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:11:39.058327 15050 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:11:39.059635 15050 hybrid_clock.cc:648] HybridClock initialized: now 1790561499059541 us; error 57 us; skew 500 ppm
I20260928 02:11:39.060014 15050 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 02:11:39 dist-test-slave-86gv krb5kdc[14649](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561499, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.235.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:11:39.063311 15050 init.cc:377] Logged in from keytab as oryx/127.6.235.67@KRBTEST.COM (short username oryx)
I20260928 02:11:39.064771 15050 webserver.cc:460] Webserver started at http://127.6.235.67:35127/ using document root <none> and password file <none>
I20260928 02:11:39.065559 15050 fs_manager.cc:359] Metadata directory not provided
I20260928 02:11:39.065649 15050 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:11:39.065917 15050 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:11:39.068835 15050 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790561432251112-7085-0/minicluster-data/ts-2/data/instance:
uuid: "baedbfdc92b946f0a07777ec1f899494"
format_stamp: "Formatted at 2026-09-28 02:11:39 on dist-test-slave-86gv"
server_key: "05b753105438c49e5097dbb66b36a16d"
server_key_iv: "586fc71eb554e535531cf272388d1718"
server_key_version: "encryptionkey@0"
I20260928 02:11:39.069890 15050 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790561432251112-7085-0/minicluster-data/ts-2/wal/instance:
uuid: "baedbfdc92b946f0a07777ec1f899494"
format_stamp: "Formatted at 2026-09-28 02:11:39 on dist-test-slave-86gv"
server_key: "05b753105438c49e5097dbb66b36a16d"
server_key_iv: "586fc71eb554e535531cf272388d1718"
server_key_version: "encryptionkey@0"
I20260928 02:11:39.075416 15050 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260928 02:11:39.079262 15065 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:11:39.081034 15050 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 02:11:39.081305 15050 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790561432251112-7085-0/minicluster-data/ts-2/data,/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790561432251112-7085-0/minicluster-data/ts-2/wal
uuid: "baedbfdc92b946f0a07777ec1f899494"
format_stamp: "Formatted at 2026-09-28 02:11:39 on dist-test-slave-86gv"
server_key: "05b753105438c49e5097dbb66b36a16d"
server_key_iv: "586fc71eb554e535531cf272388d1718"
server_key_version: "encryptionkey@0"
I20260928 02:11:39.081594 15050 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790561432251112-7085-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790561432251112-7085-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790561432251112-7085-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:11:39.083036 14903 heartbeater.cc:503] Master 127.6.235.126:32895 was elected leader, sending a full tablet report...
I20260928 02:11:39.115716 15050 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:11:39.120818 15050 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:11:39.121162 15050 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:11:39.124229 15050 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:11:39.124441 15050 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.001s sys 0.000s
I20260928 02:11:39.124660 15050 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:11:39.124786 15050 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 28 02:11:39 dist-test-slave-86gv krb5kdc[14649](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561499, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.235.67@KRBTEST.COM for oryx/127.6.235.126@KRBTEST.COM
I20260928 02:11:39.151888 15076 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:11:39.125531 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.235.126:32895 (local address 127.6.235.67:42539)
0928 02:11:39.126421 (+ 890us) negotiation.cc:107] Waiting for socket to connect
0928 02:11:39.126472 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0928 02:11:39.128376 (+ 1904us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:11:39.129210 (+ 834us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:11:39.129224 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:11:39.130059 (+ 835us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:11:39.131040 (+ 981us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:11:39.131076 (+ 36us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:11:39.133120 (+ 2044us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:11:39.133129 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:11:39.136290 (+ 3161us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:11:39.136312 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:11:39.137470 (+ 1158us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:11:39.137481 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:11:39.137896 (+ 415us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:11:39.138911 (+ 1015us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:11:39.138937 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:11:39.143186 (+ 4249us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:11:39.147051 (+ 3865us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:11:39.147060 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:11:39.147089 (+ 29us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:11:39.147624 (+ 535us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:11:39.148015 (+ 391us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:11:39.148020 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:11:39.148024 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:11:39.148228 (+ 204us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:11:39.148883 (+ 655us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:11:39.148913 (+ 30us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:11:39.149355 (+ 442us) client_negotiation.cc:769] Sending connection context
0928 02:11:39.149751 (+ 396us) client_negotiation.cc:240] Negotiation successful
0928 02:11:39.150399 (+ 648us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":603,"thread_start_us":362,"threads_started":1}
I20260928 02:11:39.153177 15074 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:11:39.124811 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.235.67:42539 (local address 127.6.235.126:32895)
0928 02:11:39.125298 (+ 487us) server_negotiation.cc:206] Beginning negotiation
0928 02:11:39.125304 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 02:11:39.126626 (+ 1322us) server_negotiation.cc:407] Connection header received
0928 02:11:39.128680 (+ 2054us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:11:39.128686 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:11:39.128775 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:11:39.128902 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:11:39.131343 (+ 2441us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:11:39.132813 (+ 1470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:11:39.136514 (+ 3701us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:11:39.137137 (+ 623us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:11:39.137305 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:11:39.143539 (+ 6234us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:11:39.143544 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:11:39.143548 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:11:39.143584 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:11:39.146902 (+ 3318us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:11:39.147750 (+ 848us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:11:39.147755 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:11:39.147761 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:11:39.147892 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:11:39.148351 (+ 459us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:11:39.148361 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:11:39.148363 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:11:39.148743 (+ 380us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:11:39.148897 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0928 02:11:39.152517 (+ 3620us) server_negotiation.cc:299] Negotiation successful
0928 02:11:39.152841 (+ 324us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":361,"thread_start_us":249,"threads_started":1}
I20260928 02:11:39.181046 15050 rpc_server.cc:307] RPC server started. Bound to: 127.6.235.67:42843
I20260928 02:11:39.181059 15186 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.235.67:42843 every 8 connection(s)
I20260928 02:11:39.182904 15050 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790561432251112-7085-0/minicluster-data/ts-2/data/info.pb
I20260928 02:11:39.184652 15074 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:11:39.165602 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.235.67:55607 (local address 127.6.235.126:32895)
0928 02:11:39.165836 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0928 02:11:39.165846 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 02:11:39.166558 (+ 712us) server_negotiation.cc:407] Connection header received
0928 02:11:39.166816 (+ 258us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:11:39.166825 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:11:39.166941 (+ 116us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:11:39.167064 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:11:39.169151 (+ 2087us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:11:39.170485 (+ 1334us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:11:39.172881 (+ 2396us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:11:39.173698 (+ 817us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:11:39.173859 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:11:39.177230 (+ 3371us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:11:39.177238 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:11:39.177243 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:11:39.177292 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:11:39.181273 (+ 3981us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:11:39.182188 (+ 915us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:11:39.182193 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:11:39.182195 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:11:39.182331 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:11:39.182748 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:11:39.182752 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:11:39.182754 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:11:39.183216 (+ 462us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:11:39.183415 (+ 199us) server_negotiation.cc:1035] Waiting for connection context
0928 02:11:39.183981 (+ 566us) server_negotiation.cc:299] Negotiation successful
0928 02:11:39.184234 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260928 02:11:39.185031 15149 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:11:39.165707 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.235.126:32895 (local address 127.6.235.67:55607)
0928 02:11:39.166421 (+ 714us) negotiation.cc:107] Waiting for socket to connect
0928 02:11:39.166441 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0928 02:11:39.166639 (+ 198us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:11:39.167131 (+ 492us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:11:39.167138 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:11:39.168070 (+ 932us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:11:39.168876 (+ 806us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:11:39.168918 (+ 42us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:11:39.170702 (+ 1784us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:11:39.170714 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:11:39.172679 (+ 1965us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:11:39.172701 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:11:39.173997 (+ 1296us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:11:39.174004 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:11:39.174272 (+ 268us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:11:39.175124 (+ 852us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:11:39.175150 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:11:39.176923 (+ 1773us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:11:39.181461 (+ 4538us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:11:39.181466 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:11:39.181472 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:11:39.182019 (+ 547us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:11:39.182467 (+ 448us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:11:39.182475 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:11:39.182483 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:11:39.182636 (+ 153us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:11:39.183346 (+ 710us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:11:39.183354 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:11:39.183643 (+ 289us) client_negotiation.cc:769] Sending connection context
0928 02:11:39.183840 (+ 197us) client_negotiation.cc:240] Negotiation successful
0928 02:11:39.184330 (+ 490us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":582,"thread_start_us":436,"threads_started":1}
I20260928 02:11:39.185497 15187 heartbeater.cc:347] Connected to a master server at 127.6.235.126:32895
I20260928 02:11:39.186822 15187 heartbeater.cc:464] Registering TS with master...
I20260928 02:11:39.187546 15187 heartbeater.cc:511] Master 127.6.235.126:32895 requested a full tablet report, sending...
I20260928 02:11:39.188879 14694 ts_manager.cc:194] Registered new tserver with Master: baedbfdc92b946f0a07777ec1f899494 (127.6.235.67:42843)
I20260928 02:11:39.189862 14694 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.6.235.67@KRBTEST.COM'} at 127.6.235.67:42539
I20260928 02:11:39.192436 7085 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTlaUuc/build/asan/bin/kudu as pid 15050
I20260928 02:11:39.192729 7085 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790561432251112-7085-0/minicluster-data/ts-2/wal/instance
I20260928 02:11:39.193216 7085 external_mini_cluster.cc:1428] Setting key 2f9d793a7e12eeb47abdf19c411c8b47
I20260928 02:11:39.196193 7085 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 28 02:11:39 dist-test-slave-86gv krb5kdc[14649](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561499, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 28 02:11:39 dist-test-slave-86gv krb5kdc[14649](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.6.235.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 02:11:39 dist-test-slave-86gv krb5kdc[14649](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.6.235.126@KRBTEST.COM, Server not found in Kerberos database
I20260928 02:11:39.631060 15046 heartbeater.cc:503] Master 127.6.235.126:32895 was elected leader, sending a full tablet report...
I20260928 02:11:40.192572 15187 heartbeater.cc:503] Master 127.6.235.126:32895 was elected leader, sending a full tablet report...
W20260928 02:11:40.328578 15195 sasl_common.cc:323] Time spent calling sasl_client_start(): real 1.085s user 0.004s sys 0.012s
W20260928 02:11:40.330015 15074 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:11:39.237703 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55434 (local address 127.6.235.126:32895)
0928 02:11:39.237918 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0928 02:11:39.237925 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 02:11:39.238042 (+ 117us) server_negotiation.cc:407] Connection header received
0928 02:11:39.238280 (+ 238us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:11:39.238285 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:11:39.238358 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:11:39.238474 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:11:39.239813 (+ 1339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:11:39.240531 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:11:39.241919 (+ 1388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:11:39.242345 (+ 426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:11:39.242484 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:11:40.329646 (+1087162us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55434: BlockingRecv error: recv got EOF from 127.0.0.1:55434 (error 108)
Metrics: {"server-negotiator.queue_time_us":72}
Sep 28 02:11:40 dist-test-slave-86gv krb5kdc[14649](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561499, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for oryx/127.6.235.126@KRBTEST.COM
I20260928 02:11:40.349865 15074 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:11:40.334819 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55442 (local address 127.6.235.126:32895)
0928 02:11:40.335023 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0928 02:11:40.335032 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 02:11:40.335413 (+ 381us) server_negotiation.cc:407] Connection header received
0928 02:11:40.335646 (+ 233us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:11:40.335650 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:11:40.335716 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:11:40.335830 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:11:40.337166 (+ 1336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:11:40.337900 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:11:40.339164 (+ 1264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:11:40.339588 (+ 424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:11:40.339736 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:11:40.343801 (+ 4065us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:11:40.343806 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:11:40.343810 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:11:40.343847 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:11:40.346626 (+ 2779us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:11:40.347621 (+ 995us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:11:40.347626 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:11:40.347628 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:11:40.347763 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:11:40.348387 (+ 624us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:11:40.348391 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:11:40.348394 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:11:40.348718 (+ 324us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:11:40.349153 (+ 435us) server_negotiation.cc:1035] Waiting for connection context
0928 02:11:40.349406 (+ 253us) server_negotiation.cc:299] Negotiation successful
0928 02:11:40.349605 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260928 02:11:40.354607 14694 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:55442:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 02:11:40.361811 14694 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 02:11:40.401234 15210 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:11:40.375283 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39054 (local address 127.6.235.67:42843)
0928 02:11:40.391764 (+ 16481us) server_negotiation.cc:206] Beginning negotiation
0928 02:11:40.391777 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0928 02:11:40.393192 (+ 1415us) server_negotiation.cc:407] Connection header received
0928 02:11:40.393515 (+ 323us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:11:40.393527 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:11:40.393833 (+ 306us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:11:40.394011 (+ 178us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:11:40.394762 (+ 751us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:11:40.396266 (+ 1504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:11:40.398468 (+ 2202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:11:40.399995 (+ 1527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:11:40.400150 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:11:40.400367 (+ 217us) server_negotiation.cc:1035] Waiting for connection context
0928 02:11:40.400700 (+ 333us) server_negotiation.cc:299] Negotiation successful
0928 02:11:40.400887 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16255,"thread_start_us":16104,"threads_started":1}
I20260928 02:11:40.401315 15209 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:11:40.375024 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.235.67:42843 (local address 127.0.0.1:39054)
0928 02:11:40.392935 (+ 17911us) negotiation.cc:107] Waiting for socket to connect
0928 02:11:40.392968 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0928 02:11:40.393161 (+ 193us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:11:40.394093 (+ 932us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:11:40.394097 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:11:40.394121 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:11:40.394544 (+ 423us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:11:40.394557 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:11:40.396373 (+ 1816us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:11:40.396380 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:11:40.398298 (+ 1918us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:11:40.398310 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:11:40.400215 (+ 1905us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:11:40.400221 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:11:40.400417 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:11:40.400480 (+ 63us) client_negotiation.cc:769] Sending connection context
0928 02:11:40.400636 (+ 156us) client_negotiation.cc:240] Negotiation successful
0928 02:11:40.400836 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":17223,"mutex_wait_us":95,"thread_start_us":17026,"threads_started":1}
I20260928 02:11:40.401343 15208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:11:40.374822 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49468 (local address 127.6.235.65:42571)
0928 02:11:40.387319 (+ 12497us) server_negotiation.cc:206] Beginning negotiation
0928 02:11:40.387334 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0928 02:11:40.390225 (+ 2891us) server_negotiation.cc:407] Connection header received
0928 02:11:40.390376 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:11:40.390389 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:11:40.390774 (+ 385us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:11:40.391020 (+ 246us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:11:40.393736 (+ 2716us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:11:40.395422 (+ 1686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:11:40.398649 (+ 3227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:11:40.400017 (+ 1368us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:11:40.400154 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:11:40.400367 (+ 213us) server_negotiation.cc:1035] Waiting for connection context
0928 02:11:40.400869 (+ 502us) server_negotiation.cc:299] Negotiation successful
0928 02:11:40.401060 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12225,"thread_start_us":12526,"threads_started":1}
I20260928 02:11:40.402031 15213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:11:40.378378 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34518 (local address 127.6.235.66:41335)
0928 02:11:40.389381 (+ 11003us) server_negotiation.cc:206] Beginning negotiation
0928 02:11:40.389394 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0928 02:11:40.393780 (+ 4386us) server_negotiation.cc:407] Connection header received
0928 02:11:40.394026 (+ 246us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:11:40.394033 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:11:40.394264 (+ 231us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:11:40.394454 (+ 190us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:11:40.395064 (+ 610us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:11:40.396487 (+ 1423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:11:40.399092 (+ 2605us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:11:40.400560 (+ 1468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:11:40.400671 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:11:40.400864 (+ 193us) server_negotiation.cc:1035] Waiting for connection context
0928 02:11:40.401545 (+ 681us) server_negotiation.cc:299] Negotiation successful
0928 02:11:40.401740 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10784,"thread_start_us":10682,"threads_started":1}
I20260928 02:11:40.402170 15211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:11:40.374611 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.235.65:42571 (local address 127.0.0.1:49468)
0928 02:11:40.389562 (+ 14951us) negotiation.cc:107] Waiting for socket to connect
0928 02:11:40.389640 (+ 78us) client_negotiation.cc:174] Beginning negotiation
0928 02:11:40.389941 (+ 301us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:11:40.391801 (+ 1860us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:11:40.391815 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:11:40.391867 (+ 52us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:11:40.393393 (+ 1526us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:11:40.393410 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:11:40.395600 (+ 2190us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:11:40.395606 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:11:40.398412 (+ 2806us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:11:40.398433 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:11:40.400511 (+ 2078us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:11:40.400516 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:11:40.400675 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:11:40.400705 (+ 30us) client_negotiation.cc:769] Sending connection context
0928 02:11:40.401589 (+ 884us) client_negotiation.cc:240] Negotiation successful
0928 02:11:40.401818 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13627,"mutex_wait_us":90,"thread_start_us":18742,"threads_started":1}
I20260928 02:11:40.402711 15207 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:11:40.377668 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.235.66:41335 (local address 127.0.0.1:34518)
0928 02:11:40.393634 (+ 15966us) negotiation.cc:107] Waiting for socket to connect
0928 02:11:40.393650 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 02:11:40.393814 (+ 164us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:11:40.394444 (+ 630us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:11:40.394448 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:11:40.394477 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:11:40.394860 (+ 383us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:11:40.394873 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:11:40.396637 (+ 1764us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:11:40.396645 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:11:40.398907 (+ 2262us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:11:40.398922 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:11:40.401067 (+ 2145us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:11:40.401073 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:11:40.401234 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:11:40.401276 (+ 42us) client_negotiation.cc:769] Sending connection context
0928 02:11:40.402276 (+ 1000us) client_negotiation.cc:240] Negotiation successful
0928 02:11:40.402399 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15843,"thread_start_us":10422,"threads_started":1}
I20260928 02:11:40.405855 14832 tablet_service.cc:1467] Processing CreateTablet for tablet a2853ac6cc6f4b6098cb372c45004caa (DEFAULT_TABLE table=test-table [id=690391fb25da477a9263ef0b7c07d0e1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:11:40.405845 15117 tablet_service.cc:1467] Processing CreateTablet for tablet a2853ac6cc6f4b6098cb372c45004caa (DEFAULT_TABLE table=test-table [id=690391fb25da477a9263ef0b7c07d0e1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:11:40.406041 14975 tablet_service.cc:1467] Processing CreateTablet for tablet a2853ac6cc6f4b6098cb372c45004caa (DEFAULT_TABLE table=test-table [id=690391fb25da477a9263ef0b7c07d0e1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:11:40.407335 14975 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a2853ac6cc6f4b6098cb372c45004caa. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:11:40.407351 15117 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a2853ac6cc6f4b6098cb372c45004caa. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:11:40.407344 14832 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a2853ac6cc6f4b6098cb372c45004caa. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:11:40.425441 15215 tablet_bootstrap.cc:492] T a2853ac6cc6f4b6098cb372c45004caa P baedbfdc92b946f0a07777ec1f899494: Bootstrap starting.
I20260928 02:11:40.431257 15214 tablet_bootstrap.cc:492] T a2853ac6cc6f4b6098cb372c45004caa P 476becc74e6c42449c408fc8bc3380b1: Bootstrap starting.
I20260928 02:11:40.433703 15216 tablet_bootstrap.cc:492] T a2853ac6cc6f4b6098cb372c45004caa P 6e7411d0f5db463799e9b2db6669ea48: Bootstrap starting.
I20260928 02:11:40.434340 15215 tablet_bootstrap.cc:654] T a2853ac6cc6f4b6098cb372c45004caa P baedbfdc92b946f0a07777ec1f899494: Neither blocks nor log segments found. Creating new log.
I20260928 02:11:40.435442 15215 log.cc:824] T a2853ac6cc6f4b6098cb372c45004caa P baedbfdc92b946f0a07777ec1f899494: Log is configured to *not* fsync() on all Append() calls
I20260928 02:11:40.436770 15216 tablet_bootstrap.cc:654] T a2853ac6cc6f4b6098cb372c45004caa P 6e7411d0f5db463799e9b2db6669ea48: Neither blocks nor log segments found. Creating new log.
I20260928 02:11:40.437829 15216 log.cc:824] T a2853ac6cc6f4b6098cb372c45004caa P 6e7411d0f5db463799e9b2db6669ea48: Log is configured to *not* fsync() on all Append() calls
I20260928 02:11:40.438840 15215 tablet_bootstrap.cc:492] T a2853ac6cc6f4b6098cb372c45004caa P baedbfdc92b946f0a07777ec1f899494: No bootstrap required, opened a new log
I20260928 02:11:40.439128 15215 ts_tablet_manager.cc:1397] T a2853ac6cc6f4b6098cb372c45004caa P baedbfdc92b946f0a07777ec1f899494: Time spent bootstrapping tablet: real 0.014s user 0.005s sys 0.003s
I20260928 02:11:40.440445 15214 tablet_bootstrap.cc:654] T a2853ac6cc6f4b6098cb372c45004caa P 476becc74e6c42449c408fc8bc3380b1: Neither blocks nor log segments found. Creating new log.
I20260928 02:11:40.440503 15216 tablet_bootstrap.cc:492] T a2853ac6cc6f4b6098cb372c45004caa P 6e7411d0f5db463799e9b2db6669ea48: No bootstrap required, opened a new log
I20260928 02:11:40.440770 15216 ts_tablet_manager.cc:1397] T a2853ac6cc6f4b6098cb372c45004caa P 6e7411d0f5db463799e9b2db6669ea48: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260928 02:11:40.441747 15214 log.cc:824] T a2853ac6cc6f4b6098cb372c45004caa P 476becc74e6c42449c408fc8bc3380b1: Log is configured to *not* fsync() on all Append() calls
I20260928 02:11:40.447935 15214 tablet_bootstrap.cc:492] T a2853ac6cc6f4b6098cb372c45004caa P 476becc74e6c42449c408fc8bc3380b1: No bootstrap required, opened a new log
I20260928 02:11:40.448268 15214 ts_tablet_manager.cc:1397] T a2853ac6cc6f4b6098cb372c45004caa P 476becc74e6c42449c408fc8bc3380b1: Time spent bootstrapping tablet: real 0.017s user 0.009s sys 0.000s
I20260928 02:11:40.449448 15216 raft_consensus.cc:348] T a2853ac6cc6f4b6098cb372c45004caa P 6e7411d0f5db463799e9b2db6669ea48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7411d0f5db463799e9b2db6669ea48" member_type: VOTER last_known_addr { host: "127.6.235.65" port: 42571 } } peers { permanent_uuid: "baedbfdc92b946f0a07777ec1f899494" member_type: VOTER last_known_addr { host: "127.6.235.67" port: 42843 } } peers { permanent_uuid: "476becc74e6c42449c408fc8bc3380b1" member_type: VOTER last_known_addr { host: "127.6.235.66" port: 41335 } }
I20260928 02:11:40.449889 15216 raft_consensus.cc:374] T a2853ac6cc6f4b6098cb372c45004caa P 6e7411d0f5db463799e9b2db6669ea48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:11:40.450021 15216 raft_consensus.cc:729] T a2853ac6cc6f4b6098cb372c45004caa P 6e7411d0f5db463799e9b2db6669ea48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6e7411d0f5db463799e9b2db6669ea48, State: Initialized, Role: FOLLOWER
I20260928 02:11:40.450552 15215 raft_consensus.cc:348] T a2853ac6cc6f4b6098cb372c45004caa P baedbfdc92b946f0a07777ec1f899494 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7411d0f5db463799e9b2db6669ea48" member_type: VOTER last_known_addr { host: "127.6.235.65" port: 42571 } } peers { permanent_uuid: "baedbfdc92b946f0a07777ec1f899494" member_type: VOTER last_known_addr { host: "127.6.235.67" port: 42843 } } peers { permanent_uuid: "476becc74e6c42449c408fc8bc3380b1" member_type: VOTER last_known_addr { host: "127.6.235.66" port: 41335 } }
I20260928 02:11:40.450695 15216 consensus_queue.cc:260] T a2853ac6cc6f4b6098cb372c45004caa P 6e7411d0f5db463799e9b2db6669ea48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7411d0f5db463799e9b2db6669ea48" member_type: VOTER last_known_addr { host: "127.6.235.65" port: 42571 } } peers { permanent_uuid: "baedbfdc92b946f0a07777ec1f899494" member_type: VOTER last_known_addr { host: "127.6.235.67" port: 42843 } } peers { permanent_uuid: "476becc74e6c42449c408fc8bc3380b1" member_type: VOTER last_known_addr { host: "127.6.235.66" port: 41335 } }
I20260928 02:11:40.451009 15215 raft_consensus.cc:374] T a2853ac6cc6f4b6098cb372c45004caa P baedbfdc92b946f0a07777ec1f899494 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:11:40.451179 15215 raft_consensus.cc:729] T a2853ac6cc6f4b6098cb372c45004caa P baedbfdc92b946f0a07777ec1f899494 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: baedbfdc92b946f0a07777ec1f899494, State: Initialized, Role: FOLLOWER
I20260928 02:11:40.451931 15215 consensus_queue.cc:260] T a2853ac6cc6f4b6098cb372c45004caa P baedbfdc92b946f0a07777ec1f899494 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7411d0f5db463799e9b2db6669ea48" member_type: VOTER last_known_addr { host: "127.6.235.65" port: 42571 } } peers { permanent_uuid: "baedbfdc92b946f0a07777ec1f899494" member_type: VOTER last_known_addr { host: "127.6.235.67" port: 42843 } } peers { permanent_uuid: "476becc74e6c42449c408fc8bc3380b1" member_type: VOTER last_known_addr { host: "127.6.235.66" port: 41335 } }
I20260928 02:11:40.452742 15216 ts_tablet_manager.cc:1428] T a2853ac6cc6f4b6098cb372c45004caa P 6e7411d0f5db463799e9b2db6669ea48: Time spent starting tablet: real 0.012s user 0.008s sys 0.005s
I20260928 02:11:40.458413 15215 ts_tablet_manager.cc:1428] T a2853ac6cc6f4b6098cb372c45004caa P baedbfdc92b946f0a07777ec1f899494: Time spent starting tablet: real 0.019s user 0.013s sys 0.006s
I20260928 02:11:40.460637 15214 raft_consensus.cc:348] T a2853ac6cc6f4b6098cb372c45004caa P 476becc74e6c42449c408fc8bc3380b1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7411d0f5db463799e9b2db6669ea48" member_type: VOTER last_known_addr { host: "127.6.235.65" port: 42571 } } peers { permanent_uuid: "baedbfdc92b946f0a07777ec1f899494" member_type: VOTER last_known_addr { host: "127.6.235.67" port: 42843 } } peers { permanent_uuid: "476becc74e6c42449c408fc8bc3380b1" member_type: VOTER last_known_addr { host: "127.6.235.66" port: 41335 } }
I20260928 02:11:40.461190 15214 raft_consensus.cc:374] T a2853ac6cc6f4b6098cb372c45004caa P 476becc74e6c42449c408fc8bc3380b1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:11:40.461308 15214 raft_consensus.cc:729] T a2853ac6cc6f4b6098cb372c45004caa P 476becc74e6c42449c408fc8bc3380b1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 476becc74e6c42449c408fc8bc3380b1, State: Initialized, Role: FOLLOWER
I20260928 02:11:40.461877 15214 consensus_queue.cc:260] T a2853ac6cc6f4b6098cb372c45004caa P 476becc74e6c42449c408fc8bc3380b1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7411d0f5db463799e9b2db6669ea48" member_type: VOTER last_known_addr { host: "127.6.235.65" port: 42571 } } peers { permanent_uuid: "baedbfdc92b946f0a07777ec1f899494" member_type: VOTER last_known_addr { host: "127.6.235.67" port: 42843 } } peers { permanent_uuid: "476becc74e6c42449c408fc8bc3380b1" member_type: VOTER last_known_addr { host: "127.6.235.66" port: 41335 } }
I20260928 02:11:40.463552 15214 ts_tablet_manager.cc:1428] T a2853ac6cc6f4b6098cb372c45004caa P 476becc74e6c42449c408fc8bc3380b1: Time spent starting tablet: real 0.015s user 0.014s sys 0.000s
I20260928 02:11:40.464843 15221 raft_consensus.cc:482] T a2853ac6cc6f4b6098cb372c45004caa P baedbfdc92b946f0a07777ec1f899494 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 02:11:40.465159 15221 raft_consensus.cc:504] T a2853ac6cc6f4b6098cb372c45004caa P baedbfdc92b946f0a07777ec1f899494 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7411d0f5db463799e9b2db6669ea48" member_type: VOTER last_known_addr { host: "127.6.235.65" port: 42571 } } peers { permanent_uuid: "baedbfdc92b946f0a07777ec1f899494" member_type: VOTER last_known_addr { host: "127.6.235.67" port: 42843 } } peers { permanent_uuid: "476becc74e6c42449c408fc8bc3380b1" member_type: VOTER last_known_addr { host: "127.6.235.66" port: 41335 } }
I20260928 02:11:40.467360 15221 leader_election.cc:288] T a2853ac6cc6f4b6098cb372c45004caa P baedbfdc92b946f0a07777ec1f899494 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6e7411d0f5db463799e9b2db6669ea48 (127.6.235.65:42571), 476becc74e6c42449c408fc8bc3380b1 (127.6.235.66:41335)
I20260928 02:11:40.474992 15213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:11:40.468273 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.235.67:44169 (local address 127.6.235.66:41335)
0928 02:11:40.468466 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0928 02:11:40.468473 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 02:11:40.468710 (+ 237us) server_negotiation.cc:407] Connection header received
0928 02:11:40.469004 (+ 294us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:11:40.469009 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:11:40.469078 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:11:40.469193 (+ 115us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:11:40.469826 (+ 633us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:11:40.470820 (+ 994us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:11:40.472945 (+ 2125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:11:40.473981 (+ 1036us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:11:40.474120 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:11:40.474182 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0928 02:11:40.474582 (+ 400us) server_negotiation.cc:299] Negotiation successful
0928 02:11:40.474721 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260928 02:11:40.475039 15223 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:11:40.467755 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.235.66:41335 (local address 127.6.235.67:44169)
0928 02:11:40.468584 (+ 829us) negotiation.cc:107] Waiting for socket to connect
0928 02:11:40.468609 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0928 02:11:40.468778 (+ 169us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:11:40.469233 (+ 455us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:11:40.469239 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:11:40.469276 (+ 37us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:11:40.469673 (+ 397us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:11:40.469685 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:11:40.470945 (+ 1260us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:11:40.470950 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:11:40.472714 (+ 1764us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:11:40.472727 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:11:40.474202 (+ 1475us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:11:40.474209 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:11:40.474385 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:11:40.474436 (+ 51us) client_negotiation.cc:769] Sending connection context
0928 02:11:40.474532 (+ 96us) client_negotiation.cc:240] Negotiation successful
0928 02:11:40.474642 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":647,"mutex_wait_us":40,"thread_start_us":1586,"threads_started":1}
I20260928 02:11:40.475244 15208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:11:40.467826 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.235.67:59327 (local address 127.6.235.65:42571)
0928 02:11:40.468088 (+ 262us) server_negotiation.cc:206] Beginning negotiation
0928 02:11:40.468098 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 02:11:40.469490 (+ 1392us) server_negotiation.cc:407] Connection header received
0928 02:11:40.469573 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:11:40.469578 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:11:40.469694 (+ 116us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:11:40.469786 (+ 92us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:11:40.470397 (+ 611us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:11:40.471224 (+ 827us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:11:40.473425 (+ 2201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:11:40.474264 (+ 839us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:11:40.474415 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:11:40.474483 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0928 02:11:40.474881 (+ 398us) server_negotiation.cc:299] Negotiation successful
0928 02:11:40.475026 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260928 02:11:40.475570 15224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:11:40.467779 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.235.65:42571 (local address 127.6.235.67:59327)
0928 02:11:40.469266 (+ 1487us) negotiation.cc:107] Waiting for socket to connect
0928 02:11:40.469284 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0928 02:11:40.469445 (+ 161us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:11:40.469822 (+ 377us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:11:40.469826 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:11:40.469850 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:11:40.470212 (+ 362us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:11:40.470224 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:11:40.471395 (+ 1171us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:11:40.471400 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:11:40.473260 (+ 1860us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:11:40.473275 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:11:40.474482 (+ 1207us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:11:40.474487 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:11:40.474689 (+ 202us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:11:40.474723 (+ 34us) client_negotiation.cc:769] Sending connection context
0928 02:11:40.474836 (+ 113us) client_negotiation.cc:240] Negotiation successful
0928 02:11:40.474993 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1345,"thread_start_us":403,"threads_started":1}
I20260928 02:11:40.476562 14853 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a2853ac6cc6f4b6098cb372c45004caa" candidate_uuid: "baedbfdc92b946f0a07777ec1f899494" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6e7411d0f5db463799e9b2db6669ea48" is_pre_election: true
I20260928 02:11:40.476562 14996 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a2853ac6cc6f4b6098cb372c45004caa" candidate_uuid: "baedbfdc92b946f0a07777ec1f899494" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "476becc74e6c42449c408fc8bc3380b1" is_pre_election: true
I20260928 02:11:40.477222 14996 raft_consensus.cc:2454] T a2853ac6cc6f4b6098cb372c45004caa P 476becc74e6c42449c408fc8bc3380b1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate baedbfdc92b946f0a07777ec1f899494 in term 0.
I20260928 02:11:40.477222 14853 raft_consensus.cc:2454] T a2853ac6cc6f4b6098cb372c45004caa P 6e7411d0f5db463799e9b2db6669ea48 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate baedbfdc92b946f0a07777ec1f899494 in term 0.
I20260928 02:11:40.477998 15066 leader_election.cc:302] T a2853ac6cc6f4b6098cb372c45004caa P baedbfdc92b946f0a07777ec1f899494 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 476becc74e6c42449c408fc8bc3380b1, 6e7411d0f5db463799e9b2db6669ea48, baedbfdc92b946f0a07777ec1f899494; no voters:
I20260928 02:11:40.478480 15221 raft_consensus.cc:2781] T a2853ac6cc6f4b6098cb372c45004caa P baedbfdc92b946f0a07777ec1f899494 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 02:11:40.478681 15221 raft_consensus.cc:482] T a2853ac6cc6f4b6098cb372c45004caa P baedbfdc92b946f0a07777ec1f899494 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 02:11:40.478834 15221 raft_consensus.cc:3037] T a2853ac6cc6f4b6098cb372c45004caa P baedbfdc92b946f0a07777ec1f899494 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:11:40.481726 15221 raft_consensus.cc:504] T a2853ac6cc6f4b6098cb372c45004caa P baedbfdc92b946f0a07777ec1f899494 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7411d0f5db463799e9b2db6669ea48" member_type: VOTER last_known_addr { host: "127.6.235.65" port: 42571 } } peers { permanent_uuid: "baedbfdc92b946f0a07777ec1f899494" member_type: VOTER last_known_addr { host: "127.6.235.67" port: 42843 } } peers { permanent_uuid: "476becc74e6c42449c408fc8bc3380b1" member_type: VOTER last_known_addr { host: "127.6.235.66" port: 41335 } }
I20260928 02:11:40.482529 15221 leader_election.cc:288] T a2853ac6cc6f4b6098cb372c45004caa P baedbfdc92b946f0a07777ec1f899494 [CANDIDATE]: Term 1 election: Requested vote from peers 6e7411d0f5db463799e9b2db6669ea48 (127.6.235.65:42571), 476becc74e6c42449c408fc8bc3380b1 (127.6.235.66:41335)
I20260928 02:11:40.483062 14996 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a2853ac6cc6f4b6098cb372c45004caa" candidate_uuid: "baedbfdc92b946f0a07777ec1f899494" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "476becc74e6c42449c408fc8bc3380b1"
I20260928 02:11:40.483083 14853 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a2853ac6cc6f4b6098cb372c45004caa" candidate_uuid: "baedbfdc92b946f0a07777ec1f899494" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6e7411d0f5db463799e9b2db6669ea48"
I20260928 02:11:40.483297 14996 raft_consensus.cc:3037] T a2853ac6cc6f4b6098cb372c45004caa P 476becc74e6c42449c408fc8bc3380b1 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:11:40.483364 14853 raft_consensus.cc:3037] T a2853ac6cc6f4b6098cb372c45004caa P 6e7411d0f5db463799e9b2db6669ea48 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:11:40.486021 14996 raft_consensus.cc:2454] T a2853ac6cc6f4b6098cb372c45004caa P 476becc74e6c42449c408fc8bc3380b1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate baedbfdc92b946f0a07777ec1f899494 in term 1.
I20260928 02:11:40.486495 15067 leader_election.cc:302] T a2853ac6cc6f4b6098cb372c45004caa P baedbfdc92b946f0a07777ec1f899494 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 476becc74e6c42449c408fc8bc3380b1, baedbfdc92b946f0a07777ec1f899494; no voters:
I20260928 02:11:40.486541 14853 raft_consensus.cc:2454] T a2853ac6cc6f4b6098cb372c45004caa P 6e7411d0f5db463799e9b2db6669ea48 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate baedbfdc92b946f0a07777ec1f899494 in term 1.
I20260928 02:11:40.486872 15221 raft_consensus.cc:2781] T a2853ac6cc6f4b6098cb372c45004caa P baedbfdc92b946f0a07777ec1f899494 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:11:40.487365 15221 raft_consensus.cc:686] T a2853ac6cc6f4b6098cb372c45004caa P baedbfdc92b946f0a07777ec1f899494 [term 1 LEADER]: Becoming Leader. State: Replica: baedbfdc92b946f0a07777ec1f899494, State: Running, Role: LEADER
I20260928 02:11:40.487929 15221 consensus_queue.cc:237] T a2853ac6cc6f4b6098cb372c45004caa P baedbfdc92b946f0a07777ec1f899494 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7411d0f5db463799e9b2db6669ea48" member_type: VOTER last_known_addr { host: "127.6.235.65" port: 42571 } } peers { permanent_uuid: "baedbfdc92b946f0a07777ec1f899494" member_type: VOTER last_known_addr { host: "127.6.235.67" port: 42843 } } peers { permanent_uuid: "476becc74e6c42449c408fc8bc3380b1" member_type: VOTER last_known_addr { host: "127.6.235.66" port: 41335 } }
I20260928 02:11:40.492983 14693 catalog_manager.cc:5495] T a2853ac6cc6f4b6098cb372c45004caa P baedbfdc92b946f0a07777ec1f899494 reported cstate change: term changed from 0 to 1, leader changed from <none> to baedbfdc92b946f0a07777ec1f899494 (127.6.235.67). New cstate: current_term: 1 leader_uuid: "baedbfdc92b946f0a07777ec1f899494" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7411d0f5db463799e9b2db6669ea48" member_type: VOTER last_known_addr { host: "127.6.235.65" port: 42571 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "baedbfdc92b946f0a07777ec1f899494" member_type: VOTER last_known_addr { host: "127.6.235.67" port: 42843 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "476becc74e6c42449c408fc8bc3380b1" member_type: VOTER last_known_addr { host: "127.6.235.66" port: 41335 } health_report { overall_health: UNKNOWN } } }
I20260928 02:11:40.518178 15210 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:11:40.512512 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39058 (local address 127.6.235.67:42843)
0928 02:11:40.512765 (+ 253us) server_negotiation.cc:206] Beginning negotiation
0928 02:11:40.512772 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 02:11:40.512911 (+ 139us) server_negotiation.cc:407] Connection header received
0928 02:11:40.513253 (+ 342us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:11:40.513257 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:11:40.513335 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:11:40.513504 (+ 169us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 02:11:40.513895 (+ 391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:11:40.514598 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:11:40.515932 (+ 1334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:11:40.516343 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:11:40.516488 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:11:40.516994 (+ 506us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 02:11:40.517575 (+ 581us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 02:11:40.517755 (+ 180us) server_negotiation.cc:1035] Waiting for connection context
0928 02:11:40.517814 (+ 59us) server_negotiation.cc:299] Negotiation successful
0928 02:11:40.517901 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260928 02:11:40.527064 14996 raft_consensus.cc:1261] T a2853ac6cc6f4b6098cb372c45004caa P 476becc74e6c42449c408fc8bc3380b1 [term 1 FOLLOWER]: Refusing update from remote peer baedbfdc92b946f0a07777ec1f899494: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 02:11:40.527065 14853 raft_consensus.cc:1261] T a2853ac6cc6f4b6098cb372c45004caa P 6e7411d0f5db463799e9b2db6669ea48 [term 1 FOLLOWER]: Refusing update from remote peer baedbfdc92b946f0a07777ec1f899494: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 02:11:40.528368 15221 consensus_queue.cc:1035] T a2853ac6cc6f4b6098cb372c45004caa P baedbfdc92b946f0a07777ec1f899494 [LEADER]: Connected to new peer: Peer: permanent_uuid: "476becc74e6c42449c408fc8bc3380b1" member_type: VOTER last_known_addr { host: "127.6.235.66" port: 41335 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 02:11:40.528751 15225 consensus_queue.cc:1035] T a2853ac6cc6f4b6098cb372c45004caa P baedbfdc92b946f0a07777ec1f899494 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e7411d0f5db463799e9b2db6669ea48" member_type: VOTER last_known_addr { host: "127.6.235.65" port: 42571 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 02:11:40.540288 15228 mvcc.cc:205] Tried to move back new op lower bound from 7334139906147008512 to 7334139906002825216. Current Snapshot: MvccSnapshot[applied={T|T < 7334139906147008512}]
I20260928 02:11:40.549504 15230 mvcc.cc:205] Tried to move back new op lower bound from 7334139906147008512 to 7334139906002825216. Current Snapshot: MvccSnapshot[applied={T|T < 7334139906147008512}]
I20260928 02:11:40.559756 14693 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:55442:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 02:11:40.560060 14693 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:55442:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 02:11:40.563866 14693 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 03972cb2d8744214a6a4570fb5531fe0: Sending DeleteTablet for 3 replicas of tablet a2853ac6cc6f4b6098cb372c45004caa
I20260928 02:11:40.565164 15117 tablet_service.cc:1514] Processing DeleteTablet for tablet a2853ac6cc6f4b6098cb372c45004caa with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:11:40 UTC) from {username='oryx', principal='oryx/127.6.235.126@KRBTEST.COM'} at 127.0.0.1:39054
I20260928 02:11:40.565160 14832 tablet_service.cc:1514] Processing DeleteTablet for tablet a2853ac6cc6f4b6098cb372c45004caa with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:11:40 UTC) from {username='oryx', principal='oryx/127.6.235.126@KRBTEST.COM'} at 127.0.0.1:49468
I20260928 02:11:40.565219 14975 tablet_service.cc:1514] Processing DeleteTablet for tablet a2853ac6cc6f4b6098cb372c45004caa with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:11:40 UTC) from {username='oryx', principal='oryx/127.6.235.126@KRBTEST.COM'} at 127.0.0.1:34518
I20260928 02:11:40.568192 14693 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:55442:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 02:11:40.568866 14693 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 02:11:40.577435 15238 tablet_replica.cc:330] T a2853ac6cc6f4b6098cb372c45004caa P baedbfdc92b946f0a07777ec1f899494: stopping tablet replica
I20260928 02:11:40.577916 15238 raft_consensus.cc:2229] T a2853ac6cc6f4b6098cb372c45004caa P baedbfdc92b946f0a07777ec1f899494 [term 1 LEADER]: Raft consensus shutting down.
I20260928 02:11:40.578862 15240 tablet_replica.cc:330] T a2853ac6cc6f4b6098cb372c45004caa P 476becc74e6c42449c408fc8bc3380b1: stopping tablet replica
I20260928 02:11:40.579100 14831 tablet_service.cc:1467] Processing CreateTablet for tablet f4383931418149c29a30539b0094007b (DEFAULT_TABLE table=test-table [id=8495c61c86d24aed87b9e0aac1573634]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:11:40.579744 14831 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f4383931418149c29a30539b0094007b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:11:40.579329 15240 raft_consensus.cc:2229] T a2853ac6cc6f4b6098cb372c45004caa P 476becc74e6c42449c408fc8bc3380b1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 02:11:40.580816 14974 tablet_service.cc:1467] Processing CreateTablet for tablet f4383931418149c29a30539b0094007b (DEFAULT_TABLE table=test-table [id=8495c61c86d24aed87b9e0aac1573634]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:11:40.581218 15240 raft_consensus.cc:2258] T a2853ac6cc6f4b6098cb372c45004caa P 476becc74e6c42449c408fc8bc3380b1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 02:11:40.581429 14974 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f4383931418149c29a30539b0094007b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:11:40.582217 15238 raft_consensus.cc:2258] T a2853ac6cc6f4b6098cb372c45004caa P baedbfdc92b946f0a07777ec1f899494 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 02:11:40.585184 15240 ts_tablet_manager.cc:1905] T a2853ac6cc6f4b6098cb372c45004caa P 476becc74e6c42449c408fc8bc3380b1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 02:11:40.585908 15117 tablet_service.cc:1467] Processing CreateTablet for tablet f4383931418149c29a30539b0094007b (DEFAULT_TABLE table=test-table [id=8495c61c86d24aed87b9e0aac1573634]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:11:40.587277 15117 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f4383931418149c29a30539b0094007b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:11:40.589839 15214 tablet_bootstrap.cc:492] T f4383931418149c29a30539b0094007b P 476becc74e6c42449c408fc8bc3380b1: Bootstrap starting.
I20260928 02:11:40.591481 15214 tablet_bootstrap.cc:654] T f4383931418149c29a30539b0094007b P 476becc74e6c42449c408fc8bc3380b1: Neither blocks nor log segments found. Creating new log.
I20260928 02:11:40.594432 15216 tablet_bootstrap.cc:492] T f4383931418149c29a30539b0094007b P 6e7411d0f5db463799e9b2db6669ea48: Bootstrap starting.
I20260928 02:11:40.595820 15215 tablet_bootstrap.cc:492] T f4383931418149c29a30539b0094007b P baedbfdc92b946f0a07777ec1f899494: Bootstrap starting.
I20260928 02:11:40.596277 15216 tablet_bootstrap.cc:654] T f4383931418149c29a30539b0094007b P 6e7411d0f5db463799e9b2db6669ea48: Neither blocks nor log segments found. Creating new log.
I20260928 02:11:40.598078 15215 tablet_bootstrap.cc:654] T f4383931418149c29a30539b0094007b P baedbfdc92b946f0a07777ec1f899494: Neither blocks nor log segments found. Creating new log.
I20260928 02:11:40.603430 15239 tablet_replica.cc:330] T a2853ac6cc6f4b6098cb372c45004caa P 6e7411d0f5db463799e9b2db6669ea48: stopping tablet replica
I20260928 02:11:40.603993 15239 raft_consensus.cc:2229] T a2853ac6cc6f4b6098cb372c45004caa P 6e7411d0f5db463799e9b2db6669ea48 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 02:11:40.604610 15239 raft_consensus.cc:2258] T a2853ac6cc6f4b6098cb372c45004caa P 6e7411d0f5db463799e9b2db6669ea48 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 02:11:40.605268 15238 ts_tablet_manager.cc:1905] T a2853ac6cc6f4b6098cb372c45004caa P baedbfdc92b946f0a07777ec1f899494: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 02:11:40.605759 15214 tablet_bootstrap.cc:492] T f4383931418149c29a30539b0094007b P 476becc74e6c42449c408fc8bc3380b1: No bootstrap required, opened a new log
I20260928 02:11:40.605930 15214 ts_tablet_manager.cc:1397] T f4383931418149c29a30539b0094007b P 476becc74e6c42449c408fc8bc3380b1: Time spent bootstrapping tablet: real 0.016s user 0.005s sys 0.005s
I20260928 02:11:40.606775 15214 raft_consensus.cc:348] T f4383931418149c29a30539b0094007b P 476becc74e6c42449c408fc8bc3380b1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7411d0f5db463799e9b2db6669ea48" member_type: VOTER last_known_addr { host: "127.6.235.65" port: 42571 } } peers { permanent_uuid: "476becc74e6c42449c408fc8bc3380b1" member_type: VOTER last_known_addr { host: "127.6.235.66" port: 41335 } } peers { permanent_uuid: "baedbfdc92b946f0a07777ec1f899494" member_type: VOTER last_known_addr { host: "127.6.235.67" port: 42843 } }
I20260928 02:11:40.608405 15214 raft_consensus.cc:374] T f4383931418149c29a30539b0094007b P 476becc74e6c42449c408fc8bc3380b1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:11:40.608521 15214 raft_consensus.cc:729] T f4383931418149c29a30539b0094007b P 476becc74e6c42449c408fc8bc3380b1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 476becc74e6c42449c408fc8bc3380b1, State: Initialized, Role: FOLLOWER
I20260928 02:11:40.608847 15214 consensus_queue.cc:260] T f4383931418149c29a30539b0094007b P 476becc74e6c42449c408fc8bc3380b1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7411d0f5db463799e9b2db6669ea48" member_type: VOTER last_known_addr { host: "127.6.235.65" port: 42571 } } peers { permanent_uuid: "476becc74e6c42449c408fc8bc3380b1" member_type: VOTER last_known_addr { host: "127.6.235.66" port: 41335 } } peers { permanent_uuid: "baedbfdc92b946f0a07777ec1f899494" member_type: VOTER last_known_addr { host: "127.6.235.67" port: 42843 } }
I20260928 02:11:40.609494 15214 ts_tablet_manager.cc:1428] T f4383931418149c29a30539b0094007b P 476becc74e6c42449c408fc8bc3380b1: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260928 02:11:40.614838 15215 tablet_bootstrap.cc:492] T f4383931418149c29a30539b0094007b P baedbfdc92b946f0a07777ec1f899494: No bootstrap required, opened a new log
I20260928 02:11:40.615027 15215 ts_tablet_manager.cc:1397] T f4383931418149c29a30539b0094007b P baedbfdc92b946f0a07777ec1f899494: Time spent bootstrapping tablet: real 0.019s user 0.000s sys 0.011s
I20260928 02:11:40.615465 15216 tablet_bootstrap.cc:492] T f4383931418149c29a30539b0094007b P 6e7411d0f5db463799e9b2db6669ea48: No bootstrap required, opened a new log
I20260928 02:11:40.615638 15216 ts_tablet_manager.cc:1397] T f4383931418149c29a30539b0094007b P 6e7411d0f5db463799e9b2db6669ea48: Time spent bootstrapping tablet: real 0.021s user 0.003s sys 0.009s
I20260928 02:11:40.616001 15215 raft_consensus.cc:348] T f4383931418149c29a30539b0094007b P baedbfdc92b946f0a07777ec1f899494 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7411d0f5db463799e9b2db6669ea48" member_type: VOTER last_known_addr { host: "127.6.235.65" port: 42571 } } peers { permanent_uuid: "476becc74e6c42449c408fc8bc3380b1" member_type: VOTER last_known_addr { host: "127.6.235.66" port: 41335 } } peers { permanent_uuid: "baedbfdc92b946f0a07777ec1f899494" member_type: VOTER last_known_addr { host: "127.6.235.67" port: 42843 } }
I20260928 02:11:40.616250 15215 raft_consensus.cc:374] T f4383931418149c29a30539b0094007b P baedbfdc92b946f0a07777ec1f899494 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:11:40.616348 15215 raft_consensus.cc:729] T f4383931418149c29a30539b0094007b P baedbfdc92b946f0a07777ec1f899494 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: baedbfdc92b946f0a07777ec1f899494, State: Initialized, Role: FOLLOWER
I20260928 02:11:40.616497 15216 raft_consensus.cc:348] T f4383931418149c29a30539b0094007b P 6e7411d0f5db463799e9b2db6669ea48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7411d0f5db463799e9b2db6669ea48" member_type: VOTER last_known_addr { host: "127.6.235.65" port: 42571 } } peers { permanent_uuid: "476becc74e6c42449c408fc8bc3380b1" member_type: VOTER last_known_addr { host: "127.6.235.66" port: 41335 } } peers { permanent_uuid: "baedbfdc92b946f0a07777ec1f899494" member_type: VOTER last_known_addr { host: "127.6.235.67" port: 42843 } }
I20260928 02:11:40.616721 15216 raft_consensus.cc:374] T f4383931418149c29a30539b0094007b P 6e7411d0f5db463799e9b2db6669ea48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:11:40.616681 15215 consensus_queue.cc:260] T f4383931418149c29a30539b0094007b P baedbfdc92b946f0a07777ec1f899494 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7411d0f5db463799e9b2db6669ea48" member_type: VOTER last_known_addr { host: "127.6.235.65" port: 42571 } } peers { permanent_uuid: "476becc74e6c42449c408fc8bc3380b1" member_type: VOTER last_known_addr { host: "127.6.235.66" port: 41335 } } peers { permanent_uuid: "baedbfdc92b946f0a07777ec1f899494" member_type: VOTER last_known_addr { host: "127.6.235.67" port: 42843 } }
I20260928 02:11:40.616832 15216 raft_consensus.cc:729] T f4383931418149c29a30539b0094007b P 6e7411d0f5db463799e9b2db6669ea48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6e7411d0f5db463799e9b2db6669ea48, State: Initialized, Role: FOLLOWER
I20260928 02:11:40.617128 15216 consensus_queue.cc:260] T f4383931418149c29a30539b0094007b P 6e7411d0f5db463799e9b2db6669ea48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7411d0f5db463799e9b2db6669ea48" member_type: VOTER last_known_addr { host: "127.6.235.65" port: 42571 } } peers { permanent_uuid: "476becc74e6c42449c408fc8bc3380b1" member_type: VOTER last_known_addr { host: "127.6.235.66" port: 41335 } } peers { permanent_uuid: "baedbfdc92b946f0a07777ec1f899494" member_type: VOTER last_known_addr { host: "127.6.235.67" port: 42843 } }
I20260928 02:11:40.617723 15216 ts_tablet_manager.cc:1428] T f4383931418149c29a30539b0094007b P 6e7411d0f5db463799e9b2db6669ea48: Time spent starting tablet: real 0.002s user 0.000s sys 0.003s
I20260928 02:11:40.616837 15239 ts_tablet_manager.cc:1905] T a2853ac6cc6f4b6098cb372c45004caa P 6e7411d0f5db463799e9b2db6669ea48: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 02:11:40.620476 15215 ts_tablet_manager.cc:1428] T f4383931418149c29a30539b0094007b P baedbfdc92b946f0a07777ec1f899494: Time spent starting tablet: real 0.005s user 0.002s sys 0.001s
I20260928 02:11:40.624832 15240 ts_tablet_manager.cc:1918] T a2853ac6cc6f4b6098cb372c45004caa P 476becc74e6c42449c408fc8bc3380b1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 02:11:40.625034 15240 log.cc:1196] T a2853ac6cc6f4b6098cb372c45004caa P 476becc74e6c42449c408fc8bc3380b1: Deleting WAL directory at /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790561432251112-7085-0/minicluster-data/ts-1/wal/wals/a2853ac6cc6f4b6098cb372c45004caa
I20260928 02:11:40.625622 15240 ts_tablet_manager.cc:1939] T a2853ac6cc6f4b6098cb372c45004caa P 476becc74e6c42449c408fc8bc3380b1: Deleting consensus metadata
W20260928 02:11:40.625736 15047 tablet.cc:2369] T f4383931418149c29a30539b0094007b P 476becc74e6c42449c408fc8bc3380b1: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 02:11:40.630492 15238 ts_tablet_manager.cc:1918] T a2853ac6cc6f4b6098cb372c45004caa P baedbfdc92b946f0a07777ec1f899494: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 02:11:40.630690 15238 log.cc:1196] T a2853ac6cc6f4b6098cb372c45004caa P baedbfdc92b946f0a07777ec1f899494: Deleting WAL directory at /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790561432251112-7085-0/minicluster-data/ts-2/wal/wals/a2853ac6cc6f4b6098cb372c45004caa
I20260928 02:11:40.630820 14679 catalog_manager.cc:4841] TS 476becc74e6c42449c408fc8bc3380b1 (127.6.235.66:41335): tablet a2853ac6cc6f4b6098cb372c45004caa (table test-table [id=690391fb25da477a9263ef0b7c07d0e1]) successfully deleted
I20260928 02:11:40.631331 15238 ts_tablet_manager.cc:1939] T a2853ac6cc6f4b6098cb372c45004caa P baedbfdc92b946f0a07777ec1f899494: Deleting consensus metadata
I20260928 02:11:40.633229 14681 catalog_manager.cc:4841] TS baedbfdc92b946f0a07777ec1f899494 (127.6.235.67:42843): tablet a2853ac6cc6f4b6098cb372c45004caa (table test-table [id=690391fb25da477a9263ef0b7c07d0e1]) successfully deleted
I20260928 02:11:40.645629 15239 ts_tablet_manager.cc:1918] T a2853ac6cc6f4b6098cb372c45004caa P 6e7411d0f5db463799e9b2db6669ea48: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 02:11:40.645844 15239 log.cc:1196] T a2853ac6cc6f4b6098cb372c45004caa P 6e7411d0f5db463799e9b2db6669ea48: Deleting WAL directory at /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790561432251112-7085-0/minicluster-data/ts-0/wal/wals/a2853ac6cc6f4b6098cb372c45004caa
I20260928 02:11:40.646387 15239 ts_tablet_manager.cc:1939] T a2853ac6cc6f4b6098cb372c45004caa P 6e7411d0f5db463799e9b2db6669ea48: Deleting consensus metadata
I20260928 02:11:40.649382 14678 catalog_manager.cc:4841] TS 6e7411d0f5db463799e9b2db6669ea48 (127.6.235.65:42571): tablet a2853ac6cc6f4b6098cb372c45004caa (table test-table [id=690391fb25da477a9263ef0b7c07d0e1]) successfully deleted
I20260928 02:11:40.673653 15220 raft_consensus.cc:482] T f4383931418149c29a30539b0094007b P 6e7411d0f5db463799e9b2db6669ea48 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 02:11:40.673933 15220 raft_consensus.cc:504] T f4383931418149c29a30539b0094007b P 6e7411d0f5db463799e9b2db6669ea48 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7411d0f5db463799e9b2db6669ea48" member_type: VOTER last_known_addr { host: "127.6.235.65" port: 42571 } } peers { permanent_uuid: "476becc74e6c42449c408fc8bc3380b1" member_type: VOTER last_known_addr { host: "127.6.235.66" port: 41335 } } peers { permanent_uuid: "baedbfdc92b946f0a07777ec1f899494" member_type: VOTER last_known_addr { host: "127.6.235.67" port: 42843 } }
I20260928 02:11:40.675971 15220 leader_election.cc:288] T f4383931418149c29a30539b0094007b P 6e7411d0f5db463799e9b2db6669ea48 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 476becc74e6c42449c408fc8bc3380b1 (127.6.235.66:41335), baedbfdc92b946f0a07777ec1f899494 (127.6.235.67:42843)
W20260928 02:11:40.687922 15188 tablet.cc:2369] T f4383931418149c29a30539b0094007b P baedbfdc92b946f0a07777ec1f899494: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 02:11:40.692574 15210 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:11:40.677235 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.235.65:57949 (local address 127.6.235.67:42843)
0928 02:11:40.677409 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0928 02:11:40.677423 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0928 02:11:40.684324 (+ 6901us) server_negotiation.cc:407] Connection header received
0928 02:11:40.684646 (+ 322us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:11:40.684655 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:11:40.684742 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:11:40.684808 (+ 66us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:11:40.685823 (+ 1015us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:11:40.687470 (+ 1647us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:11:40.690151 (+ 2681us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:11:40.691477 (+ 1326us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:11:40.691661 (+ 184us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:11:40.691741 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0928 02:11:40.692084 (+ 343us) server_negotiation.cc:299] Negotiation successful
0928 02:11:40.692231 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260928 02:11:40.692677 15244 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:11:40.676579 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.235.67:42843 (local address 127.6.235.65:57949)
0928 02:11:40.684190 (+ 7611us) negotiation.cc:107] Waiting for socket to connect
0928 02:11:40.684221 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0928 02:11:40.684461 (+ 240us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:11:40.685075 (+ 614us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:11:40.685081 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:11:40.685110 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:11:40.685570 (+ 460us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:11:40.685585 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:11:40.687666 (+ 2081us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:11:40.687673 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:11:40.689978 (+ 2305us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:11:40.689992 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:11:40.691730 (+ 1738us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:11:40.691740 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:11:40.691912 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:11:40.691950 (+ 38us) client_negotiation.cc:769] Sending connection context
0928 02:11:40.692030 (+ 80us) client_negotiation.cc:240] Negotiation successful
0928 02:11:40.692233 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7445,"thread_start_us":7216,"threads_started":1}
I20260928 02:11:40.693893 15141 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f4383931418149c29a30539b0094007b" candidate_uuid: "6e7411d0f5db463799e9b2db6669ea48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "baedbfdc92b946f0a07777ec1f899494" is_pre_election: true
I20260928 02:11:40.694345 15213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:11:40.676838 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.235.65:52737 (local address 127.6.235.66:41335)
0928 02:11:40.677085 (+ 247us) server_negotiation.cc:206] Beginning negotiation
0928 02:11:40.677092 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 02:11:40.687778 (+ 10686us) server_negotiation.cc:407] Connection header received
0928 02:11:40.688059 (+ 281us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:11:40.688065 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:11:40.688142 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:11:40.688205 (+ 63us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:11:40.689148 (+ 943us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:11:40.690160 (+ 1012us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:11:40.691772 (+ 1612us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:11:40.693095 (+ 1323us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:11:40.693271 (+ 176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:11:40.693348 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0928 02:11:40.693953 (+ 605us) server_negotiation.cc:299] Negotiation successful
0928 02:11:40.694079 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":121}
I20260928 02:11:40.694417 15245 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:11:40.676600 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.235.66:41335 (local address 127.6.235.65:52737)
0928 02:11:40.687645 (+ 11045us) negotiation.cc:107] Waiting for socket to connect
0928 02:11:40.687671 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0928 02:11:40.687891 (+ 220us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:11:40.688408 (+ 517us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:11:40.688425 (+ 17us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:11:40.688452 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:11:40.688936 (+ 484us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:11:40.688949 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:11:40.690295 (+ 1346us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:11:40.690302 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:11:40.691576 (+ 1274us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:11:40.691592 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:11:40.693573 (+ 1981us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:11:40.693579 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:11:40.693764 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:11:40.693787 (+ 23us) client_negotiation.cc:769] Sending connection context
0928 02:11:40.693927 (+ 140us) client_negotiation.cc:240] Negotiation successful
0928 02:11:40.694056 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10801,"mutex_wait_us":72,"thread_start_us":10503,"threads_started":1}
I20260928 02:11:40.694375 15141 raft_consensus.cc:2454] T f4383931418149c29a30539b0094007b P baedbfdc92b946f0a07777ec1f899494 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e7411d0f5db463799e9b2db6669ea48 in term 0.
I20260928 02:11:40.695086 14785 leader_election.cc:302] T f4383931418149c29a30539b0094007b P 6e7411d0f5db463799e9b2db6669ea48 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6e7411d0f5db463799e9b2db6669ea48, baedbfdc92b946f0a07777ec1f899494; no voters:
I20260928 02:11:40.695241 14996 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f4383931418149c29a30539b0094007b" candidate_uuid: "6e7411d0f5db463799e9b2db6669ea48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "476becc74e6c42449c408fc8bc3380b1" is_pre_election: true
I20260928 02:11:40.695466 14996 raft_consensus.cc:2454] T f4383931418149c29a30539b0094007b P 476becc74e6c42449c408fc8bc3380b1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e7411d0f5db463799e9b2db6669ea48 in term 0.
I20260928 02:11:40.695945 15220 raft_consensus.cc:2781] T f4383931418149c29a30539b0094007b P 6e7411d0f5db463799e9b2db6669ea48 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 02:11:40.696153 15220 raft_consensus.cc:482] T f4383931418149c29a30539b0094007b P 6e7411d0f5db463799e9b2db6669ea48 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 02:11:40.696241 15220 raft_consensus.cc:3037] T f4383931418149c29a30539b0094007b P 6e7411d0f5db463799e9b2db6669ea48 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:11:40.699476 15220 raft_consensus.cc:504] T f4383931418149c29a30539b0094007b P 6e7411d0f5db463799e9b2db6669ea48 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7411d0f5db463799e9b2db6669ea48" member_type: VOTER last_known_addr { host: "127.6.235.65" port: 42571 } } peers { permanent_uuid: "476becc74e6c42449c408fc8bc3380b1" member_type: VOTER last_known_addr { host: "127.6.235.66" port: 41335 } } peers { permanent_uuid: "baedbfdc92b946f0a07777ec1f899494" member_type: VOTER last_known_addr { host: "127.6.235.67" port: 42843 } }
I20260928 02:11:40.700760 14996 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f4383931418149c29a30539b0094007b" candidate_uuid: "6e7411d0f5db463799e9b2db6669ea48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "476becc74e6c42449c408fc8bc3380b1"
I20260928 02:11:40.700816 15141 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f4383931418149c29a30539b0094007b" candidate_uuid: "6e7411d0f5db463799e9b2db6669ea48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "baedbfdc92b946f0a07777ec1f899494"
I20260928 02:11:40.701025 14996 raft_consensus.cc:3037] T f4383931418149c29a30539b0094007b P 476becc74e6c42449c408fc8bc3380b1 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:11:40.701050 15141 raft_consensus.cc:3037] T f4383931418149c29a30539b0094007b P baedbfdc92b946f0a07777ec1f899494 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:11:40.701731 15220 leader_election.cc:288] T f4383931418149c29a30539b0094007b P 6e7411d0f5db463799e9b2db6669ea48 [CANDIDATE]: Term 1 election: Requested vote from peers 476becc74e6c42449c408fc8bc3380b1 (127.6.235.66:41335), baedbfdc92b946f0a07777ec1f899494 (127.6.235.67:42843)
I20260928 02:11:40.704092 14996 raft_consensus.cc:2454] T f4383931418149c29a30539b0094007b P 476becc74e6c42449c408fc8bc3380b1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6e7411d0f5db463799e9b2db6669ea48 in term 1.
I20260928 02:11:40.704253 15141 raft_consensus.cc:2454] T f4383931418149c29a30539b0094007b P baedbfdc92b946f0a07777ec1f899494 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6e7411d0f5db463799e9b2db6669ea48 in term 1.
I20260928 02:11:40.704681 14783 leader_election.cc:302] T f4383931418149c29a30539b0094007b P 6e7411d0f5db463799e9b2db6669ea48 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 476becc74e6c42449c408fc8bc3380b1, 6e7411d0f5db463799e9b2db6669ea48; no voters:
I20260928 02:11:40.705060 15220 raft_consensus.cc:2781] T f4383931418149c29a30539b0094007b P 6e7411d0f5db463799e9b2db6669ea48 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:11:40.705619 15220 raft_consensus.cc:686] T f4383931418149c29a30539b0094007b P 6e7411d0f5db463799e9b2db6669ea48 [term 1 LEADER]: Becoming Leader. State: Replica: 6e7411d0f5db463799e9b2db6669ea48, State: Running, Role: LEADER
I20260928 02:11:40.706034 15220 consensus_queue.cc:237] T f4383931418149c29a30539b0094007b P 6e7411d0f5db463799e9b2db6669ea48 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7411d0f5db463799e9b2db6669ea48" member_type: VOTER last_known_addr { host: "127.6.235.65" port: 42571 } } peers { permanent_uuid: "476becc74e6c42449c408fc8bc3380b1" member_type: VOTER last_known_addr { host: "127.6.235.66" port: 41335 } } peers { permanent_uuid: "baedbfdc92b946f0a07777ec1f899494" member_type: VOTER last_known_addr { host: "127.6.235.67" port: 42843 } }
I20260928 02:11:40.709650 14693 catalog_manager.cc:5495] T f4383931418149c29a30539b0094007b P 6e7411d0f5db463799e9b2db6669ea48 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6e7411d0f5db463799e9b2db6669ea48 (127.6.235.65). New cstate: current_term: 1 leader_uuid: "6e7411d0f5db463799e9b2db6669ea48" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7411d0f5db463799e9b2db6669ea48" member_type: VOTER last_known_addr { host: "127.6.235.65" port: 42571 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "476becc74e6c42449c408fc8bc3380b1" member_type: VOTER last_known_addr { host: "127.6.235.66" port: 41335 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "baedbfdc92b946f0a07777ec1f899494" member_type: VOTER last_known_addr { host: "127.6.235.67" port: 42843 } health_report { overall_health: UNKNOWN } } }
Sep 28 02:11:40 dist-test-slave-86gv krb5kdc[14649](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561497, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.235.126@KRBTEST.COM for oryx/127.6.235.126@KRBTEST.COM
I20260928 02:11:40.771234 15074 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:11:40.755845 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55452 (local address 127.6.235.126:32895)
0928 02:11:40.756183 (+ 338us) server_negotiation.cc:206] Beginning negotiation
0928 02:11:40.756193 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 02:11:40.756873 (+ 680us) server_negotiation.cc:407] Connection header received
0928 02:11:40.757129 (+ 256us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:11:40.757135 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:11:40.757207 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:11:40.757315 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:11:40.758546 (+ 1231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:11:40.759336 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:11:40.760709 (+ 1373us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:11:40.761239 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:11:40.761392 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:11:40.765055 (+ 3663us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:11:40.765060 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:11:40.765063 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:11:40.765095 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:11:40.768077 (+ 2982us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:11:40.768889 (+ 812us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:11:40.768929 (+ 40us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:11:40.768936 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:11:40.769072 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:11:40.769522 (+ 450us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:11:40.769529 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:11:40.769533 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:11:40.769828 (+ 295us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:11:40.770007 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0928 02:11:40.770455 (+ 448us) server_negotiation.cc:299] Negotiation successful
0928 02:11:40.770842 (+ 387us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":153,"server-negotiator.queue_time_us":78}
I20260928 02:11:40.771289 15253 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:11:40.755954 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.235.126:32895 (local address 127.0.0.1:55452)
0928 02:11:40.756745 (+ 791us) negotiation.cc:107] Waiting for socket to connect
0928 02:11:40.756789 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0928 02:11:40.756986 (+ 197us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:11:40.757337 (+ 351us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:11:40.757341 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:11:40.758106 (+ 765us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:11:40.758385 (+ 279us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:11:40.758394 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:11:40.759494 (+ 1100us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:11:40.759500 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:11:40.760554 (+ 1054us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:11:40.760565 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:11:40.761432 (+ 867us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:11:40.761440 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:11:40.761633 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:11:40.762430 (+ 797us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:11:40.762447 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:11:40.764803 (+ 2356us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:11:40.768257 (+ 3454us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:11:40.768272 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:11:40.768282 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:11:40.768752 (+ 470us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:11:40.769203 (+ 451us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:11:40.769208 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:11:40.769212 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:11:40.769383 (+ 171us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:11:40.769978 (+ 595us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:11:40.769987 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:11:40.770164 (+ 177us) client_negotiation.cc:769] Sending connection context
0928 02:11:40.770354 (+ 190us) client_negotiation.cc:240] Negotiation successful
0928 02:11:40.770695 (+ 341us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":627,"thread_start_us":389,"threads_started":1}
I20260928 02:11:40.777086 14693 catalog_manager.cc:1908] Servicing CreateTable request from {username='oryx', principal='oryx/127.6.235.126@KRBTEST.COM'} at 127.0.0.1:55452:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260928 02:11:40.778220 14693 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 02:11:40.786198 14832 tablet_service.cc:1467] Processing CreateTablet for tablet f9b85cdcf8c04a46a8679c95bda17ac6 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=b6fd0cdb52fd43bfbab2ff729db976bf]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260928 02:11:40.786442 15117 tablet_service.cc:1467] Processing CreateTablet for tablet f9b85cdcf8c04a46a8679c95bda17ac6 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=b6fd0cdb52fd43bfbab2ff729db976bf]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260928 02:11:40.786895 14832 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f9b85cdcf8c04a46a8679c95bda17ac6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:11:40.787130 15117 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f9b85cdcf8c04a46a8679c95bda17ac6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:11:40.792388 15216 tablet_bootstrap.cc:492] T f9b85cdcf8c04a46a8679c95bda17ac6 P 6e7411d0f5db463799e9b2db6669ea48: Bootstrap starting.
I20260928 02:11:40.793499 14974 tablet_service.cc:1467] Processing CreateTablet for tablet f9b85cdcf8c04a46a8679c95bda17ac6 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=b6fd0cdb52fd43bfbab2ff729db976bf]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260928 02:11:40.794150 14974 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f9b85cdcf8c04a46a8679c95bda17ac6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:11:40.795051 15215 tablet_bootstrap.cc:492] T f9b85cdcf8c04a46a8679c95bda17ac6 P baedbfdc92b946f0a07777ec1f899494: Bootstrap starting.
I20260928 02:11:40.795481 15216 tablet_bootstrap.cc:654] T f9b85cdcf8c04a46a8679c95bda17ac6 P 6e7411d0f5db463799e9b2db6669ea48: Neither blocks nor log segments found. Creating new log.
I20260928 02:11:40.796792 15215 tablet_bootstrap.cc:654] T f9b85cdcf8c04a46a8679c95bda17ac6 P baedbfdc92b946f0a07777ec1f899494: Neither blocks nor log segments found. Creating new log.
I20260928 02:11:40.804234 15214 tablet_bootstrap.cc:492] T f9b85cdcf8c04a46a8679c95bda17ac6 P 476becc74e6c42449c408fc8bc3380b1: Bootstrap starting.
I20260928 02:11:40.806114 15214 tablet_bootstrap.cc:654] T f9b85cdcf8c04a46a8679c95bda17ac6 P 476becc74e6c42449c408fc8bc3380b1: Neither blocks nor log segments found. Creating new log.
I20260928 02:11:40.806635 15215 tablet_bootstrap.cc:492] T f9b85cdcf8c04a46a8679c95bda17ac6 P baedbfdc92b946f0a07777ec1f899494: No bootstrap required, opened a new log
I20260928 02:11:40.806811 15215 ts_tablet_manager.cc:1397] T f9b85cdcf8c04a46a8679c95bda17ac6 P baedbfdc92b946f0a07777ec1f899494: Time spent bootstrapping tablet: real 0.012s user 0.008s sys 0.001s
I20260928 02:11:40.807627 15215 raft_consensus.cc:348] T f9b85cdcf8c04a46a8679c95bda17ac6 P baedbfdc92b946f0a07777ec1f899494 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7411d0f5db463799e9b2db6669ea48" member_type: VOTER last_known_addr { host: "127.6.235.65" port: 42571 } } peers { permanent_uuid: "476becc74e6c42449c408fc8bc3380b1" member_type: VOTER last_known_addr { host: "127.6.235.66" port: 41335 } } peers { permanent_uuid: "baedbfdc92b946f0a07777ec1f899494" member_type: VOTER last_known_addr { host: "127.6.235.67" port: 42843 } }
I20260928 02:11:40.807886 15215 raft_consensus.cc:374] T f9b85cdcf8c04a46a8679c95bda17ac6 P baedbfdc92b946f0a07777ec1f899494 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:11:40.807988 15215 raft_consensus.cc:729] T f9b85cdcf8c04a46a8679c95bda17ac6 P baedbfdc92b946f0a07777ec1f899494 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: baedbfdc92b946f0a07777ec1f899494, State: Initialized, Role: FOLLOWER
I20260928 02:11:40.808234 15215 consensus_queue.cc:260] T f9b85cdcf8c04a46a8679c95bda17ac6 P baedbfdc92b946f0a07777ec1f899494 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7411d0f5db463799e9b2db6669ea48" member_type: VOTER last_known_addr { host: "127.6.235.65" port: 42571 } } peers { permanent_uuid: "476becc74e6c42449c408fc8bc3380b1" member_type: VOTER last_known_addr { host: "127.6.235.66" port: 41335 } } peers { permanent_uuid: "baedbfdc92b946f0a07777ec1f899494" member_type: VOTER last_known_addr { host: "127.6.235.67" port: 42843 } }
I20260928 02:11:40.808682 15236 tablet_replica.cc:439] T f9b85cdcf8c04a46a8679c95bda17ac6 P baedbfdc92b946f0a07777ec1f899494: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7411d0f5db463799e9b2db6669ea48" member_type: VOTER last_known_addr { host: "127.6.235.65" port: 42571 } } peers { permanent_uuid: "476becc74e6c42449c408fc8bc3380b1" member_type: VOTER last_known_addr { host: "127.6.235.66" port: 41335 } } peers { permanent_uuid: "baedbfdc92b946f0a07777ec1f899494" member_type: VOTER last_known_addr { host: "127.6.235.67" port: 42843 } } }
I20260928 02:11:40.809115 15236 tablet_replica.cc:442] T f9b85cdcf8c04a46a8679c95bda17ac6 P baedbfdc92b946f0a07777ec1f899494: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 02:11:40.810340 15215 ts_tablet_manager.cc:1428] T f9b85cdcf8c04a46a8679c95bda17ac6 P baedbfdc92b946f0a07777ec1f899494: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20260928 02:11:40.810853 15214 tablet_bootstrap.cc:492] T f9b85cdcf8c04a46a8679c95bda17ac6 P 476becc74e6c42449c408fc8bc3380b1: No bootstrap required, opened a new log
I20260928 02:11:40.811023 15214 ts_tablet_manager.cc:1397] T f9b85cdcf8c04a46a8679c95bda17ac6 P 476becc74e6c42449c408fc8bc3380b1: Time spent bootstrapping tablet: real 0.007s user 0.000s sys 0.005s
I20260928 02:11:40.811736 15214 raft_consensus.cc:348] T f9b85cdcf8c04a46a8679c95bda17ac6 P 476becc74e6c42449c408fc8bc3380b1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7411d0f5db463799e9b2db6669ea48" member_type: VOTER last_known_addr { host: "127.6.235.65" port: 42571 } } peers { permanent_uuid: "476becc74e6c42449c408fc8bc3380b1" member_type: VOTER last_known_addr { host: "127.6.235.66" port: 41335 } } peers { permanent_uuid: "baedbfdc92b946f0a07777ec1f899494" member_type: VOTER last_known_addr { host: "127.6.235.67" port: 42843 } }
I20260928 02:11:40.811988 15214 raft_consensus.cc:374] T f9b85cdcf8c04a46a8679c95bda17ac6 P 476becc74e6c42449c408fc8bc3380b1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:11:40.812068 15214 raft_consensus.cc:729] T f9b85cdcf8c04a46a8679c95bda17ac6 P 476becc74e6c42449c408fc8bc3380b1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 476becc74e6c42449c408fc8bc3380b1, State: Initialized, Role: FOLLOWER
I20260928 02:11:40.812299 15214 consensus_queue.cc:260] T f9b85cdcf8c04a46a8679c95bda17ac6 P 476becc74e6c42449c408fc8bc3380b1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7411d0f5db463799e9b2db6669ea48" member_type: VOTER last_known_addr { host: "127.6.235.65" port: 42571 } } peers { permanent_uuid: "476becc74e6c42449c408fc8bc3380b1" member_type: VOTER last_known_addr { host: "127.6.235.66" port: 41335 } } peers { permanent_uuid: "baedbfdc92b946f0a07777ec1f899494" member_type: VOTER last_known_addr { host: "127.6.235.67" port: 42843 } }
I20260928 02:11:40.812920 15214 ts_tablet_manager.cc:1428] T f9b85cdcf8c04a46a8679c95bda17ac6 P 476becc74e6c42449c408fc8bc3380b1: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260928 02:11:40.813138 15216 tablet_bootstrap.cc:492] T f9b85cdcf8c04a46a8679c95bda17ac6 P 6e7411d0f5db463799e9b2db6669ea48: No bootstrap required, opened a new log
I20260928 02:11:40.813498 15216 ts_tablet_manager.cc:1397] T f9b85cdcf8c04a46a8679c95bda17ac6 P 6e7411d0f5db463799e9b2db6669ea48: Time spent bootstrapping tablet: real 0.021s user 0.009s sys 0.002s
I20260928 02:11:40.813844 15222 tablet_replica.cc:439] T f9b85cdcf8c04a46a8679c95bda17ac6 P 476becc74e6c42449c408fc8bc3380b1: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7411d0f5db463799e9b2db6669ea48" member_type: VOTER last_known_addr { host: "127.6.235.65" port: 42571 } } peers { permanent_uuid: "476becc74e6c42449c408fc8bc3380b1" member_type: VOTER last_known_addr { host: "127.6.235.66" port: 41335 } } peers { permanent_uuid: "baedbfdc92b946f0a07777ec1f899494" member_type: VOTER last_known_addr { host: "127.6.235.67" port: 42843 } } }
I20260928 02:11:40.814201 15222 tablet_replica.cc:442] T f9b85cdcf8c04a46a8679c95bda17ac6 P 476becc74e6c42449c408fc8bc3380b1: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 02:11:40.815541 15216 raft_consensus.cc:348] T f9b85cdcf8c04a46a8679c95bda17ac6 P 6e7411d0f5db463799e9b2db6669ea48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7411d0f5db463799e9b2db6669ea48" member_type: VOTER last_known_addr { host: "127.6.235.65" port: 42571 } } peers { permanent_uuid: "476becc74e6c42449c408fc8bc3380b1" member_type: VOTER last_known_addr { host: "127.6.235.66" port: 41335 } } peers { permanent_uuid: "baedbfdc92b946f0a07777ec1f899494" member_type: VOTER last_known_addr { host: "127.6.235.67" port: 42843 } }
I20260928 02:11:40.815812 15216 raft_consensus.cc:374] T f9b85cdcf8c04a46a8679c95bda17ac6 P 6e7411d0f5db463799e9b2db6669ea48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:11:40.815907 15216 raft_consensus.cc:729] T f9b85cdcf8c04a46a8679c95bda17ac6 P 6e7411d0f5db463799e9b2db6669ea48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6e7411d0f5db463799e9b2db6669ea48, State: Initialized, Role: FOLLOWER
I20260928 02:11:40.816169 15216 consensus_queue.cc:260] T f9b85cdcf8c04a46a8679c95bda17ac6 P 6e7411d0f5db463799e9b2db6669ea48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7411d0f5db463799e9b2db6669ea48" member_type: VOTER last_known_addr { host: "127.6.235.65" port: 42571 } } peers { permanent_uuid: "476becc74e6c42449c408fc8bc3380b1" member_type: VOTER last_known_addr { host: "127.6.235.66" port: 41335 } } peers { permanent_uuid: "baedbfdc92b946f0a07777ec1f899494" member_type: VOTER last_known_addr { host: "127.6.235.67" port: 42843 } }
I20260928 02:11:40.816612 15220 tablet_replica.cc:439] T f9b85cdcf8c04a46a8679c95bda17ac6 P 6e7411d0f5db463799e9b2db6669ea48: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7411d0f5db463799e9b2db6669ea48" member_type: VOTER last_known_addr { host: "127.6.235.65" port: 42571 } } peers { permanent_uuid: "476becc74e6c42449c408fc8bc3380b1" member_type: VOTER last_known_addr { host: "127.6.235.66" port: 41335 } } peers { permanent_uuid: "baedbfdc92b946f0a07777ec1f899494" member_type: VOTER last_known_addr { host: "127.6.235.67" port: 42843 } } }
I20260928 02:11:40.816851 15216 ts_tablet_manager.cc:1428] T f9b85cdcf8c04a46a8679c95bda17ac6 P 6e7411d0f5db463799e9b2db6669ea48: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20260928 02:11:40.816970 15220 tablet_replica.cc:442] T f9b85cdcf8c04a46a8679c95bda17ac6 P 6e7411d0f5db463799e9b2db6669ea48: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 02:11:40.823640 15236 raft_consensus.cc:482] T f9b85cdcf8c04a46a8679c95bda17ac6 P baedbfdc92b946f0a07777ec1f899494 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 02:11:40.823899 15236 raft_consensus.cc:504] T f9b85cdcf8c04a46a8679c95bda17ac6 P baedbfdc92b946f0a07777ec1f899494 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7411d0f5db463799e9b2db6669ea48" member_type: VOTER last_known_addr { host: "127.6.235.65" port: 42571 } } peers { permanent_uuid: "476becc74e6c42449c408fc8bc3380b1" member_type: VOTER last_known_addr { host: "127.6.235.66" port: 41335 } } peers { permanent_uuid: "baedbfdc92b946f0a07777ec1f899494" member_type: VOTER last_known_addr { host: "127.6.235.67" port: 42843 } }
I20260928 02:11:40.824740 15236 leader_election.cc:288] T f9b85cdcf8c04a46a8679c95bda17ac6 P baedbfdc92b946f0a07777ec1f899494 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6e7411d0f5db463799e9b2db6669ea48 (127.6.235.65:42571), 476becc74e6c42449c408fc8bc3380b1 (127.6.235.66:41335)
I20260928 02:11:40.825254 14853 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f9b85cdcf8c04a46a8679c95bda17ac6" candidate_uuid: "baedbfdc92b946f0a07777ec1f899494" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6e7411d0f5db463799e9b2db6669ea48" is_pre_election: true
I20260928 02:11:40.825551 14853 raft_consensus.cc:2454] T f9b85cdcf8c04a46a8679c95bda17ac6 P 6e7411d0f5db463799e9b2db6669ea48 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate baedbfdc92b946f0a07777ec1f899494 in term 0.
I20260928 02:11:40.825510 14996 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f9b85cdcf8c04a46a8679c95bda17ac6" candidate_uuid: "baedbfdc92b946f0a07777ec1f899494" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "476becc74e6c42449c408fc8bc3380b1" is_pre_election: true
I20260928 02:11:40.825776 14996 raft_consensus.cc:2454] T f9b85cdcf8c04a46a8679c95bda17ac6 P 476becc74e6c42449c408fc8bc3380b1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate baedbfdc92b946f0a07777ec1f899494 in term 0.
I20260928 02:11:40.825946 15066 leader_election.cc:302] T f9b85cdcf8c04a46a8679c95bda17ac6 P baedbfdc92b946f0a07777ec1f899494 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6e7411d0f5db463799e9b2db6669ea48, baedbfdc92b946f0a07777ec1f899494; no voters:
W20260928 02:11:40.826174 14904 tablet.cc:2369] T f4383931418149c29a30539b0094007b P 6e7411d0f5db463799e9b2db6669ea48: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 02:11:40.826333 15236 raft_consensus.cc:2781] T f9b85cdcf8c04a46a8679c95bda17ac6 P baedbfdc92b946f0a07777ec1f899494 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 02:11:40.826444 15236 raft_consensus.cc:482] T f9b85cdcf8c04a46a8679c95bda17ac6 P baedbfdc92b946f0a07777ec1f899494 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 02:11:40.826555 15236 raft_consensus.cc:3037] T f9b85cdcf8c04a46a8679c95bda17ac6 P baedbfdc92b946f0a07777ec1f899494 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:11:40.828861 15236 raft_consensus.cc:504] T f9b85cdcf8c04a46a8679c95bda17ac6 P baedbfdc92b946f0a07777ec1f899494 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7411d0f5db463799e9b2db6669ea48" member_type: VOTER last_known_addr { host: "127.6.235.65" port: 42571 } } peers { permanent_uuid: "476becc74e6c42449c408fc8bc3380b1" member_type: VOTER last_known_addr { host: "127.6.235.66" port: 41335 } } peers { permanent_uuid: "baedbfdc92b946f0a07777ec1f899494" member_type: VOTER last_known_addr { host: "127.6.235.67" port: 42843 } }
I20260928 02:11:40.829633 15236 leader_election.cc:288] T f9b85cdcf8c04a46a8679c95bda17ac6 P baedbfdc92b946f0a07777ec1f899494 [CANDIDATE]: Term 1 election: Requested vote from peers 6e7411d0f5db463799e9b2db6669ea48 (127.6.235.65:42571), 476becc74e6c42449c408fc8bc3380b1 (127.6.235.66:41335)
I20260928 02:11:40.830214 14853 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f9b85cdcf8c04a46a8679c95bda17ac6" candidate_uuid: "baedbfdc92b946f0a07777ec1f899494" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6e7411d0f5db463799e9b2db6669ea48"
I20260928 02:11:40.830356 14996 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f9b85cdcf8c04a46a8679c95bda17ac6" candidate_uuid: "baedbfdc92b946f0a07777ec1f899494" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "476becc74e6c42449c408fc8bc3380b1"
I20260928 02:11:40.830456 14853 raft_consensus.cc:3037] T f9b85cdcf8c04a46a8679c95bda17ac6 P 6e7411d0f5db463799e9b2db6669ea48 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:11:40.830611 14996 raft_consensus.cc:3037] T f9b85cdcf8c04a46a8679c95bda17ac6 P 476becc74e6c42449c408fc8bc3380b1 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:11:40.833115 14853 raft_consensus.cc:2454] T f9b85cdcf8c04a46a8679c95bda17ac6 P 6e7411d0f5db463799e9b2db6669ea48 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate baedbfdc92b946f0a07777ec1f899494 in term 1.
I20260928 02:11:40.833590 14996 raft_consensus.cc:2454] T f9b85cdcf8c04a46a8679c95bda17ac6 P 476becc74e6c42449c408fc8bc3380b1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate baedbfdc92b946f0a07777ec1f899494 in term 1.
I20260928 02:11:40.833628 15066 leader_election.cc:302] T f9b85cdcf8c04a46a8679c95bda17ac6 P baedbfdc92b946f0a07777ec1f899494 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6e7411d0f5db463799e9b2db6669ea48, baedbfdc92b946f0a07777ec1f899494; no voters:
I20260928 02:11:40.833997 15236 raft_consensus.cc:2781] T f9b85cdcf8c04a46a8679c95bda17ac6 P baedbfdc92b946f0a07777ec1f899494 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:11:40.834192 15236 raft_consensus.cc:686] T f9b85cdcf8c04a46a8679c95bda17ac6 P baedbfdc92b946f0a07777ec1f899494 [term 1 LEADER]: Becoming Leader. State: Replica: baedbfdc92b946f0a07777ec1f899494, State: Running, Role: LEADER
I20260928 02:11:40.834496 15236 consensus_queue.cc:237] T f9b85cdcf8c04a46a8679c95bda17ac6 P baedbfdc92b946f0a07777ec1f899494 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7411d0f5db463799e9b2db6669ea48" member_type: VOTER last_known_addr { host: "127.6.235.65" port: 42571 } } peers { permanent_uuid: "476becc74e6c42449c408fc8bc3380b1" member_type: VOTER last_known_addr { host: "127.6.235.66" port: 41335 } } peers { permanent_uuid: "baedbfdc92b946f0a07777ec1f899494" member_type: VOTER last_known_addr { host: "127.6.235.67" port: 42843 } }
I20260928 02:11:40.836540 15221 tablet_replica.cc:439] T f9b85cdcf8c04a46a8679c95bda17ac6 P baedbfdc92b946f0a07777ec1f899494: TxnStatusTablet state changed. Reason: New leader baedbfdc92b946f0a07777ec1f899494. Latest consensus state: current_term: 1 leader_uuid: "baedbfdc92b946f0a07777ec1f899494" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7411d0f5db463799e9b2db6669ea48" member_type: VOTER last_known_addr { host: "127.6.235.65" port: 42571 } } peers { permanent_uuid: "476becc74e6c42449c408fc8bc3380b1" member_type: VOTER last_known_addr { host: "127.6.235.66" port: 41335 } } peers { permanent_uuid: "baedbfdc92b946f0a07777ec1f899494" member_type: VOTER last_known_addr { host: "127.6.235.67" port: 42843 } } }
I20260928 02:11:40.836867 15221 tablet_replica.cc:442] T f9b85cdcf8c04a46a8679c95bda17ac6 P baedbfdc92b946f0a07777ec1f899494: This TxnStatusTablet replica's current role is: LEADER
I20260928 02:11:40.838490 14693 catalog_manager.cc:5495] T f9b85cdcf8c04a46a8679c95bda17ac6 P baedbfdc92b946f0a07777ec1f899494 reported cstate change: term changed from 0 to 1, leader changed from <none> to baedbfdc92b946f0a07777ec1f899494 (127.6.235.67). New cstate: current_term: 1 leader_uuid: "baedbfdc92b946f0a07777ec1f899494" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7411d0f5db463799e9b2db6669ea48" member_type: VOTER last_known_addr { host: "127.6.235.65" port: 42571 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "476becc74e6c42449c408fc8bc3380b1" member_type: VOTER last_known_addr { host: "127.6.235.66" port: 41335 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "baedbfdc92b946f0a07777ec1f899494" member_type: VOTER last_known_addr { host: "127.6.235.67" port: 42843 } health_report { overall_health: HEALTHY } } }
I20260928 02:11:40.841463 15258 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260928 02:11:40.841805 15258 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260928 02:11:40.865214 15253 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:11:40.859282 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.235.67:42843 (local address 127.0.0.1:39070)
0928 02:11:40.859503 (+ 221us) negotiation.cc:107] Waiting for socket to connect
0928 02:11:40.859520 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0928 02:11:40.859628 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:11:40.860035 (+ 407us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:11:40.860039 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:11:40.860049 (+ 10us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 02:11:40.860218 (+ 169us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:11:40.860226 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:11:40.861734 (+ 1508us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:11:40.861738 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:11:40.862846 (+ 1108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:11:40.862859 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:11:40.863909 (+ 1050us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:11:40.863915 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:11:40.864121 (+ 206us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:11:40.864182 (+ 61us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 02:11:40.864628 (+ 446us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 02:11:40.864654 (+ 26us) client_negotiation.cc:769] Sending connection context
0928 02:11:40.864756 (+ 102us) client_negotiation.cc:240] Negotiation successful
0928 02:11:40.864884 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":33}
I20260928 02:11:40.865216 15210 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:11:40.859423 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39070 (local address 127.6.235.67:42843)
0928 02:11:40.859642 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0928 02:11:40.859650 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 02:11:40.859679 (+ 29us) server_negotiation.cc:407] Connection header received
0928 02:11:40.859761 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:11:40.859766 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:11:40.859851 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:11:40.859959 (+ 108us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 02:11:40.860368 (+ 409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:11:40.861594 (+ 1226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:11:40.863018 (+ 1424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:11:40.863740 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:11:40.863871 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:11:40.864368 (+ 497us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 02:11:40.864516 (+ 148us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 02:11:40.864615 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0928 02:11:40.864780 (+ 165us) server_negotiation.cc:299] Negotiation successful
0928 02:11:40.864872 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260928 02:11:40.869199 14853 raft_consensus.cc:1261] T f9b85cdcf8c04a46a8679c95bda17ac6 P 6e7411d0f5db463799e9b2db6669ea48 [term 1 FOLLOWER]: Refusing update from remote peer baedbfdc92b946f0a07777ec1f899494: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 02:11:40.869253 14996 raft_consensus.cc:1261] T f9b85cdcf8c04a46a8679c95bda17ac6 P 476becc74e6c42449c408fc8bc3380b1 [term 1 FOLLOWER]: Refusing update from remote peer baedbfdc92b946f0a07777ec1f899494: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 02:11:40.869808 15221 consensus_queue.cc:1035] T f9b85cdcf8c04a46a8679c95bda17ac6 P baedbfdc92b946f0a07777ec1f899494 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e7411d0f5db463799e9b2db6669ea48" member_type: VOTER last_known_addr { host: "127.6.235.65" port: 42571 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 02:11:40.870078 15236 consensus_queue.cc:1035] T f9b85cdcf8c04a46a8679c95bda17ac6 P baedbfdc92b946f0a07777ec1f899494 [LEADER]: Connected to new peer: Peer: permanent_uuid: "476becc74e6c42449c408fc8bc3380b1" member_type: VOTER last_known_addr { host: "127.6.235.66" port: 41335 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 02:11:40.873426 15222 tablet_replica.cc:439] T f9b85cdcf8c04a46a8679c95bda17ac6 P 476becc74e6c42449c408fc8bc3380b1: TxnStatusTablet state changed. Reason: New leader baedbfdc92b946f0a07777ec1f899494. Latest consensus state: current_term: 1 leader_uuid: "baedbfdc92b946f0a07777ec1f899494" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7411d0f5db463799e9b2db6669ea48" member_type: VOTER last_known_addr { host: "127.6.235.65" port: 42571 } } peers { permanent_uuid: "476becc74e6c42449c408fc8bc3380b1" member_type: VOTER last_known_addr { host: "127.6.235.66" port: 41335 } } peers { permanent_uuid: "baedbfdc92b946f0a07777ec1f899494" member_type: VOTER last_known_addr { host: "127.6.235.67" port: 42843 } } }
I20260928 02:11:40.873759 15222 tablet_replica.cc:442] T f9b85cdcf8c04a46a8679c95bda17ac6 P 476becc74e6c42449c408fc8bc3380b1: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 02:11:40.875044 15220 tablet_replica.cc:439] T f9b85cdcf8c04a46a8679c95bda17ac6 P 6e7411d0f5db463799e9b2db6669ea48: TxnStatusTablet state changed. Reason: New leader baedbfdc92b946f0a07777ec1f899494. Latest consensus state: current_term: 1 leader_uuid: "baedbfdc92b946f0a07777ec1f899494" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7411d0f5db463799e9b2db6669ea48" member_type: VOTER last_known_addr { host: "127.6.235.65" port: 42571 } } peers { permanent_uuid: "476becc74e6c42449c408fc8bc3380b1" member_type: VOTER last_known_addr { host: "127.6.235.66" port: 41335 } } peers { permanent_uuid: "baedbfdc92b946f0a07777ec1f899494" member_type: VOTER last_known_addr { host: "127.6.235.67" port: 42843 } } }
I20260928 02:11:40.875361 15220 tablet_replica.cc:442] T f9b85cdcf8c04a46a8679c95bda17ac6 P 6e7411d0f5db463799e9b2db6669ea48: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 02:11:40.876121 15225 tablet_replica.cc:439] T f9b85cdcf8c04a46a8679c95bda17ac6 P baedbfdc92b946f0a07777ec1f899494: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "baedbfdc92b946f0a07777ec1f899494" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7411d0f5db463799e9b2db6669ea48" member_type: VOTER last_known_addr { host: "127.6.235.65" port: 42571 } } peers { permanent_uuid: "476becc74e6c42449c408fc8bc3380b1" member_type: VOTER last_known_addr { host: "127.6.235.66" port: 41335 } } peers { permanent_uuid: "baedbfdc92b946f0a07777ec1f899494" member_type: VOTER last_known_addr { host: "127.6.235.67" port: 42843 } } }
I20260928 02:11:40.879334 15222 tablet_replica.cc:439] T f9b85cdcf8c04a46a8679c95bda17ac6 P 476becc74e6c42449c408fc8bc3380b1: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "baedbfdc92b946f0a07777ec1f899494" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7411d0f5db463799e9b2db6669ea48" member_type: VOTER last_known_addr { host: "127.6.235.65" port: 42571 } } peers { permanent_uuid: "476becc74e6c42449c408fc8bc3380b1" member_type: VOTER last_known_addr { host: "127.6.235.66" port: 41335 } } peers { permanent_uuid: "baedbfdc92b946f0a07777ec1f899494" member_type: VOTER last_known_addr { host: "127.6.235.67" port: 42843 } } }
I20260928 02:11:40.879583 15222 tablet_replica.cc:442] T f9b85cdcf8c04a46a8679c95bda17ac6 P 476becc74e6c42449c408fc8bc3380b1: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 02:11:40.881950 15221 tablet_replica.cc:439] T f9b85cdcf8c04a46a8679c95bda17ac6 P baedbfdc92b946f0a07777ec1f899494: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "baedbfdc92b946f0a07777ec1f899494" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7411d0f5db463799e9b2db6669ea48" member_type: VOTER last_known_addr { host: "127.6.235.65" port: 42571 } } peers { permanent_uuid: "476becc74e6c42449c408fc8bc3380b1" member_type: VOTER last_known_addr { host: "127.6.235.66" port: 41335 } } peers { permanent_uuid: "baedbfdc92b946f0a07777ec1f899494" member_type: VOTER last_known_addr { host: "127.6.235.67" port: 42843 } } }
I20260928 02:11:40.883252 15220 tablet_replica.cc:439] T f9b85cdcf8c04a46a8679c95bda17ac6 P 6e7411d0f5db463799e9b2db6669ea48: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "baedbfdc92b946f0a07777ec1f899494" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7411d0f5db463799e9b2db6669ea48" member_type: VOTER last_known_addr { host: "127.6.235.65" port: 42571 } } peers { permanent_uuid: "476becc74e6c42449c408fc8bc3380b1" member_type: VOTER last_known_addr { host: "127.6.235.66" port: 41335 } } peers { permanent_uuid: "baedbfdc92b946f0a07777ec1f899494" member_type: VOTER last_known_addr { host: "127.6.235.67" port: 42843 } } }
I20260928 02:11:40.883670 15220 tablet_replica.cc:442] T f9b85cdcf8c04a46a8679c95bda17ac6 P 6e7411d0f5db463799e9b2db6669ea48: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 02:11:40.884480 15225 tablet_replica.cc:442] T f9b85cdcf8c04a46a8679c95bda17ac6 P baedbfdc92b946f0a07777ec1f899494: This TxnStatusTablet replica's current role is: LEADER
I20260928 02:11:40.890339 15221 tablet_replica.cc:442] T f9b85cdcf8c04a46a8679c95bda17ac6 P baedbfdc92b946f0a07777ec1f899494: This TxnStatusTablet replica's current role is: LEADER
I20260928 02:11:40.900103 15208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:11:40.894569 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49474 (local address 127.6.235.65:42571)
0928 02:11:40.894761 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0928 02:11:40.894768 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 02:11:40.894903 (+ 135us) server_negotiation.cc:407] Connection header received
0928 02:11:40.895160 (+ 257us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:11:40.895164 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:11:40.895241 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:11:40.895390 (+ 149us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 02:11:40.895776 (+ 386us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:11:40.896528 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:11:40.898002 (+ 1474us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:11:40.898398 (+ 396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:11:40.898552 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:11:40.898972 (+ 420us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 02:11:40.899503 (+ 531us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 02:11:40.899675 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0928 02:11:40.899722 (+ 47us) server_negotiation.cc:299] Negotiation successful
0928 02:11:40.899803 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260928 02:11:40.918597 15210 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:11:40.911781 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.235.65:50017 (local address 127.6.235.67:42843)
0928 02:11:40.912006 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0928 02:11:40.912012 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 02:11:40.912581 (+ 569us) server_negotiation.cc:407] Connection header received
0928 02:11:40.912804 (+ 223us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:11:40.912809 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:11:40.912923 (+ 114us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:11:40.913062 (+ 139us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 02:11:40.913635 (+ 573us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:11:40.914813 (+ 1178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:11:40.916316 (+ 1503us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:11:40.917048 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:11:40.917189 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:11:40.917637 (+ 448us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 02:11:40.917851 (+ 214us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 02:11:40.917956 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0928 02:11:40.918160 (+ 204us) server_negotiation.cc:299] Negotiation successful
0928 02:11:40.918256 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260928 02:11:40.918669 15266 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:11:40.911675 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.235.67:42843 (local address 127.6.235.65:50017)
0928 02:11:40.912290 (+ 615us) negotiation.cc:107] Waiting for socket to connect
0928 02:11:40.912456 (+ 166us) client_negotiation.cc:174] Beginning negotiation
0928 02:11:40.912650 (+ 194us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:11:40.913126 (+ 476us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:11:40.913131 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:11:40.913145 (+ 14us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 02:11:40.913472 (+ 327us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:11:40.913482 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:11:40.915027 (+ 1545us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:11:40.915031 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:11:40.916157 (+ 1126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:11:40.916169 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:11:40.917244 (+ 1075us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:11:40.917249 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:11:40.917381 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:11:40.917452 (+ 71us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 02:11:40.918019 (+ 567us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 02:11:40.918029 (+ 10us) client_negotiation.cc:769] Sending connection context
0928 02:11:40.918099 (+ 70us) client_negotiation.cc:240] Negotiation successful
0928 02:11:40.918293 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":462,"thread_start_us":305,"threads_started":1}
I20260928 02:11:40.928722 14996 raft_consensus.cc:1261] T f4383931418149c29a30539b0094007b P 476becc74e6c42449c408fc8bc3380b1 [term 1 FOLLOWER]: Refusing update from remote peer 6e7411d0f5db463799e9b2db6669ea48: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 02:11:40.929118 15141 raft_consensus.cc:1261] T f4383931418149c29a30539b0094007b P baedbfdc92b946f0a07777ec1f899494 [term 1 FOLLOWER]: Refusing update from remote peer 6e7411d0f5db463799e9b2db6669ea48: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 02:11:40.929427 15246 consensus_queue.cc:1035] T f4383931418149c29a30539b0094007b P 6e7411d0f5db463799e9b2db6669ea48 [LEADER]: Connected to new peer: Peer: permanent_uuid: "476becc74e6c42449c408fc8bc3380b1" member_type: VOTER last_known_addr { host: "127.6.235.66" port: 41335 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 02:11:40.929924 15220 consensus_queue.cc:1035] T f4383931418149c29a30539b0094007b P 6e7411d0f5db463799e9b2db6669ea48 [LEADER]: Connected to new peer: Peer: permanent_uuid: "baedbfdc92b946f0a07777ec1f899494" member_type: VOTER last_known_addr { host: "127.6.235.67" port: 42843 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 02:11:40.943780 15229 mvcc.cc:205] Tried to move back new op lower bound from 7334139907797864448 to 7334139906895757312. Current Snapshot: MvccSnapshot[applied={T|T < 7334139907797864448}]
I20260928 02:11:40.975529 15208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:11:40.967893 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.235.67:34933 (local address 127.6.235.65:42571)
0928 02:11:40.968212 (+ 319us) server_negotiation.cc:206] Beginning negotiation
0928 02:11:40.968221 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 02:11:40.968819 (+ 598us) server_negotiation.cc:407] Connection header received
0928 02:11:40.969110 (+ 291us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:11:40.969123 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:11:40.969194 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:11:40.969297 (+ 103us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 02:11:40.970124 (+ 827us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:11:40.971190 (+ 1066us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:11:40.973219 (+ 2029us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:11:40.973904 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:11:40.974054 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:11:40.974659 (+ 605us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 02:11:40.974803 (+ 144us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 02:11:40.974905 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0928 02:11:40.975112 (+ 207us) server_negotiation.cc:299] Negotiation successful
0928 02:11:40.975205 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":161}
I20260928 02:11:40.975783 15270 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:11:40.967907 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.235.65:42571 (local address 127.6.235.67:34933)
0928 02:11:40.968688 (+ 781us) negotiation.cc:107] Waiting for socket to connect
0928 02:11:40.968715 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0928 02:11:40.968908 (+ 193us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:11:40.969330 (+ 422us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:11:40.969336 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:11:40.969350 (+ 14us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 02:11:40.969780 (+ 430us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:11:40.969790 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:11:40.971340 (+ 1550us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:11:40.971347 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:11:40.973053 (+ 1706us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:11:40.973068 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:11:40.974146 (+ 1078us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:11:40.974152 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:11:40.974323 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:11:40.974427 (+ 104us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 02:11:40.974939 (+ 512us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 02:11:40.974962 (+ 23us) client_negotiation.cc:769] Sending connection context
0928 02:11:40.975065 (+ 103us) client_negotiation.cc:240] Negotiation successful
0928 02:11:40.975242 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":602,"thread_start_us":411,"threads_started":1}
I20260928 02:11:41.021811 14694 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:55442:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 02:11:41.022100 14694 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:55442:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 02:11:41.025914 14694 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 03972cb2d8744214a6a4570fb5531fe0: Sending DeleteTablet for 3 replicas of tablet f4383931418149c29a30539b0094007b
I20260928 02:11:41.026947 14974 tablet_service.cc:1514] Processing DeleteTablet for tablet f4383931418149c29a30539b0094007b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:11:41 UTC) from {username='oryx', principal='oryx/127.6.235.126@KRBTEST.COM'} at 127.0.0.1:34518
I20260928 02:11:41.026937 14832 tablet_service.cc:1514] Processing DeleteTablet for tablet f4383931418149c29a30539b0094007b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:11:41 UTC) from {username='oryx', principal='oryx/127.6.235.126@KRBTEST.COM'} at 127.0.0.1:49468
I20260928 02:11:41.027254 15240 tablet_replica.cc:330] T f4383931418149c29a30539b0094007b P 476becc74e6c42449c408fc8bc3380b1: stopping tablet replica
I20260928 02:11:41.027256 15239 tablet_replica.cc:330] T f4383931418149c29a30539b0094007b P 6e7411d0f5db463799e9b2db6669ea48: stopping tablet replica
I20260928 02:11:41.027484 15239 raft_consensus.cc:2229] T f4383931418149c29a30539b0094007b P 6e7411d0f5db463799e9b2db6669ea48 [term 1 LEADER]: Raft consensus shutting down.
I20260928 02:11:41.027496 15240 raft_consensus.cc:2229] T f4383931418149c29a30539b0094007b P 476becc74e6c42449c408fc8bc3380b1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 02:11:41.027695 15240 raft_consensus.cc:2258] T f4383931418149c29a30539b0094007b P 476becc74e6c42449c408fc8bc3380b1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 02:11:41.027951 15117 tablet_service.cc:1514] Processing DeleteTablet for tablet f4383931418149c29a30539b0094007b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:11:41 UTC) from {username='oryx', principal='oryx/127.6.235.126@KRBTEST.COM'} at 127.0.0.1:39054
I20260928 02:11:41.028002 15239 raft_consensus.cc:2258] T f4383931418149c29a30539b0094007b P 6e7411d0f5db463799e9b2db6669ea48 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 02:11:41.028501 15238 tablet_replica.cc:330] T f4383931418149c29a30539b0094007b P baedbfdc92b946f0a07777ec1f899494: stopping tablet replica
I20260928 02:11:41.028713 15238 raft_consensus.cc:2229] T f4383931418149c29a30539b0094007b P baedbfdc92b946f0a07777ec1f899494 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 02:11:41.028955 15238 raft_consensus.cc:2258] T f4383931418149c29a30539b0094007b P baedbfdc92b946f0a07777ec1f899494 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 02:11:41.034286 15240 ts_tablet_manager.cc:1905] T f4383931418149c29a30539b0094007b P 476becc74e6c42449c408fc8bc3380b1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 02:11:41.042138 15239 ts_tablet_manager.cc:1905] T f4383931418149c29a30539b0094007b P 6e7411d0f5db463799e9b2db6669ea48: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 02:11:41.042239 15238 ts_tablet_manager.cc:1905] T f4383931418149c29a30539b0094007b P baedbfdc92b946f0a07777ec1f899494: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 02:11:41.050074 15239 ts_tablet_manager.cc:1918] T f4383931418149c29a30539b0094007b P 6e7411d0f5db463799e9b2db6669ea48: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 02:11:41.050264 15239 log.cc:1196] T f4383931418149c29a30539b0094007b P 6e7411d0f5db463799e9b2db6669ea48: Deleting WAL directory at /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790561432251112-7085-0/minicluster-data/ts-0/wal/wals/f4383931418149c29a30539b0094007b
I20260928 02:11:41.050704 15239 ts_tablet_manager.cc:1939] T f4383931418149c29a30539b0094007b P 6e7411d0f5db463799e9b2db6669ea48: Deleting consensus metadata
I20260928 02:11:41.051489 15240 ts_tablet_manager.cc:1918] T f4383931418149c29a30539b0094007b P 476becc74e6c42449c408fc8bc3380b1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 02:11:41.051926 14678 catalog_manager.cc:4841] TS 6e7411d0f5db463799e9b2db6669ea48 (127.6.235.65:42571): tablet f4383931418149c29a30539b0094007b (table test-table [id=8495c61c86d24aed87b9e0aac1573634]) successfully deleted
I20260928 02:11:41.052253 15240 log.cc:1196] T f4383931418149c29a30539b0094007b P 476becc74e6c42449c408fc8bc3380b1: Deleting WAL directory at /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790561432251112-7085-0/minicluster-data/ts-1/wal/wals/f4383931418149c29a30539b0094007b
I20260928 02:11:41.055050 15240 ts_tablet_manager.cc:1939] T f4383931418149c29a30539b0094007b P 476becc74e6c42449c408fc8bc3380b1: Deleting consensus metadata
I20260928 02:11:41.055728 15238 ts_tablet_manager.cc:1918] T f4383931418149c29a30539b0094007b P baedbfdc92b946f0a07777ec1f899494: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 02:11:41.055922 15238 log.cc:1196] T f4383931418149c29a30539b0094007b P baedbfdc92b946f0a07777ec1f899494: Deleting WAL directory at /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790561432251112-7085-0/minicluster-data/ts-2/wal/wals/f4383931418149c29a30539b0094007b
I20260928 02:11:41.056386 15238 ts_tablet_manager.cc:1939] T f4383931418149c29a30539b0094007b P baedbfdc92b946f0a07777ec1f899494: Deleting consensus metadata
I20260928 02:11:41.057436 14679 catalog_manager.cc:4841] TS 476becc74e6c42449c408fc8bc3380b1 (127.6.235.66:41335): tablet f4383931418149c29a30539b0094007b (table test-table [id=8495c61c86d24aed87b9e0aac1573634]) successfully deleted
I20260928 02:11:41.058789 14681 catalog_manager.cc:4841] TS baedbfdc92b946f0a07777ec1f899494 (127.6.235.67:42843): tablet f4383931418149c29a30539b0094007b (table test-table [id=8495c61c86d24aed87b9e0aac1573634]) successfully deleted
Sep 28 02:11:41 dist-test-slave-86gv krb5kdc[14649](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561501, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 28 02:11:41 dist-test-slave-86gv krb5kdc[14649](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.6.235.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 02:11:41 dist-test-slave-86gv krb5kdc[14649](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.6.235.126@KRBTEST.COM, Server not found in Kerberos database
W20260928 02:11:41.134356 15074 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:11:41.119866 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55462 (local address 127.6.235.126:32895)
0928 02:11:41.120103 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0928 02:11:41.120110 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 02:11:41.120763 (+ 653us) server_negotiation.cc:407] Connection header received
0928 02:11:41.121022 (+ 259us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:11:41.121028 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:11:41.121115 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:11:41.121224 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:11:41.122960 (+ 1736us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:11:41.124137 (+ 1177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:11:41.126178 (+ 2041us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:11:41.126857 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:11:41.126995 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:11:41.134075 (+ 7080us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55462: BlockingRecv error: recv got EOF from 127.0.0.1:55462 (error 108)
Metrics: {"server-negotiator.queue_time_us":80}
Sep 28 02:11:41 dist-test-slave-86gv krb5kdc[14649](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561501, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.6.235.126@KRBTEST.COM
I20260928 02:11:41.162045 15074 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:11:41.142592 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55466 (local address 127.6.235.126:32895)
0928 02:11:41.142797 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0928 02:11:41.142804 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 02:11:41.143403 (+ 599us) server_negotiation.cc:407] Connection header received
0928 02:11:41.143633 (+ 230us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:11:41.143639 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:11:41.143737 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:11:41.143864 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:11:41.145590 (+ 1726us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:11:41.146793 (+ 1203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:11:41.148828 (+ 2035us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:11:41.149524 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:11:41.149792 (+ 268us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:11:41.155077 (+ 5285us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:11:41.155085 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:11:41.155090 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:11:41.155150 (+ 60us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:11:41.158846 (+ 3696us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:11:41.159764 (+ 918us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:11:41.159770 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:11:41.159774 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:11:41.159948 (+ 174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:11:41.160376 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:11:41.160381 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:11:41.160385 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:11:41.160804 (+ 419us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:11:41.161103 (+ 299us) server_negotiation.cc:1035] Waiting for connection context
0928 02:11:41.161414 (+ 311us) server_negotiation.cc:299] Negotiation successful
0928 02:11:41.161659 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260928 02:11:41.166532 14694 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55466:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 02:11:41.167284 14694 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 02:11:41.176286 14832 tablet_service.cc:1467] Processing CreateTablet for tablet 3239999622d94b1196274986003cfb39 (DEFAULT_TABLE table=test-table [id=ebade711c2ea4302a98c82d42686c0e0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:11:41.176870 14832 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3239999622d94b1196274986003cfb39. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:11:41.179482 14974 tablet_service.cc:1467] Processing CreateTablet for tablet 3239999622d94b1196274986003cfb39 (DEFAULT_TABLE table=test-table [id=ebade711c2ea4302a98c82d42686c0e0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:11:41.180032 14974 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3239999622d94b1196274986003cfb39. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:11:41.184190 15216 tablet_bootstrap.cc:492] T 3239999622d94b1196274986003cfb39 P 6e7411d0f5db463799e9b2db6669ea48: Bootstrap starting.
I20260928 02:11:41.186195 15216 tablet_bootstrap.cc:654] T 3239999622d94b1196274986003cfb39 P 6e7411d0f5db463799e9b2db6669ea48: Neither blocks nor log segments found. Creating new log.
I20260928 02:11:41.187924 15117 tablet_service.cc:1467] Processing CreateTablet for tablet 3239999622d94b1196274986003cfb39 (DEFAULT_TABLE table=test-table [id=ebade711c2ea4302a98c82d42686c0e0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:11:41.188434 15117 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3239999622d94b1196274986003cfb39. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:11:41.196455 15214 tablet_bootstrap.cc:492] T 3239999622d94b1196274986003cfb39 P 476becc74e6c42449c408fc8bc3380b1: Bootstrap starting.
I20260928 02:11:41.198693 15214 tablet_bootstrap.cc:654] T 3239999622d94b1196274986003cfb39 P 476becc74e6c42449c408fc8bc3380b1: Neither blocks nor log segments found. Creating new log.
I20260928 02:11:41.198925 15215 tablet_bootstrap.cc:492] T 3239999622d94b1196274986003cfb39 P baedbfdc92b946f0a07777ec1f899494: Bootstrap starting.
I20260928 02:11:41.201082 15215 tablet_bootstrap.cc:654] T 3239999622d94b1196274986003cfb39 P baedbfdc92b946f0a07777ec1f899494: Neither blocks nor log segments found. Creating new log.
I20260928 02:11:41.217804 15215 tablet_bootstrap.cc:492] T 3239999622d94b1196274986003cfb39 P baedbfdc92b946f0a07777ec1f899494: No bootstrap required, opened a new log
I20260928 02:11:41.218179 15215 ts_tablet_manager.cc:1397] T 3239999622d94b1196274986003cfb39 P baedbfdc92b946f0a07777ec1f899494: Time spent bootstrapping tablet: real 0.019s user 0.009s sys 0.001s
I20260928 02:11:41.219236 15216 tablet_bootstrap.cc:492] T 3239999622d94b1196274986003cfb39 P 6e7411d0f5db463799e9b2db6669ea48: No bootstrap required, opened a new log
I20260928 02:11:41.219136 15215 raft_consensus.cc:348] T 3239999622d94b1196274986003cfb39 P baedbfdc92b946f0a07777ec1f899494 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7411d0f5db463799e9b2db6669ea48" member_type: VOTER last_known_addr { host: "127.6.235.65" port: 42571 } } peers { permanent_uuid: "476becc74e6c42449c408fc8bc3380b1" member_type: VOTER last_known_addr { host: "127.6.235.66" port: 41335 } } peers { permanent_uuid: "baedbfdc92b946f0a07777ec1f899494" member_type: VOTER last_known_addr { host: "127.6.235.67" port: 42843 } }
I20260928 02:11:41.219414 15216 ts_tablet_manager.cc:1397] T 3239999622d94b1196274986003cfb39 P 6e7411d0f5db463799e9b2db6669ea48: Time spent bootstrapping tablet: real 0.035s user 0.006s sys 0.005s
I20260928 02:11:41.219419 15215 raft_consensus.cc:374] T 3239999622d94b1196274986003cfb39 P baedbfdc92b946f0a07777ec1f899494 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:11:41.219523 15215 raft_consensus.cc:729] T 3239999622d94b1196274986003cfb39 P baedbfdc92b946f0a07777ec1f899494 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: baedbfdc92b946f0a07777ec1f899494, State: Initialized, Role: FOLLOWER
I20260928 02:11:41.219789 15215 consensus_queue.cc:260] T 3239999622d94b1196274986003cfb39 P baedbfdc92b946f0a07777ec1f899494 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7411d0f5db463799e9b2db6669ea48" member_type: VOTER last_known_addr { host: "127.6.235.65" port: 42571 } } peers { permanent_uuid: "476becc74e6c42449c408fc8bc3380b1" member_type: VOTER last_known_addr { host: "127.6.235.66" port: 41335 } } peers { permanent_uuid: "baedbfdc92b946f0a07777ec1f899494" member_type: VOTER last_known_addr { host: "127.6.235.67" port: 42843 } }
I20260928 02:11:41.220358 15215 ts_tablet_manager.cc:1428] T 3239999622d94b1196274986003cfb39 P baedbfdc92b946f0a07777ec1f899494: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20260928 02:11:41.220405 15216 raft_consensus.cc:348] T 3239999622d94b1196274986003cfb39 P 6e7411d0f5db463799e9b2db6669ea48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7411d0f5db463799e9b2db6669ea48" member_type: VOTER last_known_addr { host: "127.6.235.65" port: 42571 } } peers { permanent_uuid: "476becc74e6c42449c408fc8bc3380b1" member_type: VOTER last_known_addr { host: "127.6.235.66" port: 41335 } } peers { permanent_uuid: "baedbfdc92b946f0a07777ec1f899494" member_type: VOTER last_known_addr { host: "127.6.235.67" port: 42843 } }
I20260928 02:11:41.220711 15216 raft_consensus.cc:374] T 3239999622d94b1196274986003cfb39 P 6e7411d0f5db463799e9b2db6669ea48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:11:41.220835 15216 raft_consensus.cc:729] T 3239999622d94b1196274986003cfb39 P 6e7411d0f5db463799e9b2db6669ea48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6e7411d0f5db463799e9b2db6669ea48, State: Initialized, Role: FOLLOWER
I20260928 02:11:41.221194 15216 consensus_queue.cc:260] T 3239999622d94b1196274986003cfb39 P 6e7411d0f5db463799e9b2db6669ea48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7411d0f5db463799e9b2db6669ea48" member_type: VOTER last_known_addr { host: "127.6.235.65" port: 42571 } } peers { permanent_uuid: "476becc74e6c42449c408fc8bc3380b1" member_type: VOTER last_known_addr { host: "127.6.235.66" port: 41335 } } peers { permanent_uuid: "baedbfdc92b946f0a07777ec1f899494" member_type: VOTER last_known_addr { host: "127.6.235.67" port: 42843 } }
I20260928 02:11:41.225953 15216 ts_tablet_manager.cc:1428] T 3239999622d94b1196274986003cfb39 P 6e7411d0f5db463799e9b2db6669ea48: Time spent starting tablet: real 0.006s user 0.003s sys 0.000s
I20260928 02:11:41.230391 15214 tablet_bootstrap.cc:492] T 3239999622d94b1196274986003cfb39 P 476becc74e6c42449c408fc8bc3380b1: No bootstrap required, opened a new log
I20260928 02:11:41.230615 15214 ts_tablet_manager.cc:1397] T 3239999622d94b1196274986003cfb39 P 476becc74e6c42449c408fc8bc3380b1: Time spent bootstrapping tablet: real 0.034s user 0.004s sys 0.012s
I20260928 02:11:41.231458 15214 raft_consensus.cc:348] T 3239999622d94b1196274986003cfb39 P 476becc74e6c42449c408fc8bc3380b1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7411d0f5db463799e9b2db6669ea48" member_type: VOTER last_known_addr { host: "127.6.235.65" port: 42571 } } peers { permanent_uuid: "476becc74e6c42449c408fc8bc3380b1" member_type: VOTER last_known_addr { host: "127.6.235.66" port: 41335 } } peers { permanent_uuid: "baedbfdc92b946f0a07777ec1f899494" member_type: VOTER last_known_addr { host: "127.6.235.67" port: 42843 } }
I20260928 02:11:41.231748 15214 raft_consensus.cc:374] T 3239999622d94b1196274986003cfb39 P 476becc74e6c42449c408fc8bc3380b1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:11:41.231900 15214 raft_consensus.cc:729] T 3239999622d94b1196274986003cfb39 P 476becc74e6c42449c408fc8bc3380b1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 476becc74e6c42449c408fc8bc3380b1, State: Initialized, Role: FOLLOWER
I20260928 02:11:41.232144 15214 consensus_queue.cc:260] T 3239999622d94b1196274986003cfb39 P 476becc74e6c42449c408fc8bc3380b1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7411d0f5db463799e9b2db6669ea48" member_type: VOTER last_known_addr { host: "127.6.235.65" port: 42571 } } peers { permanent_uuid: "476becc74e6c42449c408fc8bc3380b1" member_type: VOTER last_known_addr { host: "127.6.235.66" port: 41335 } } peers { permanent_uuid: "baedbfdc92b946f0a07777ec1f899494" member_type: VOTER last_known_addr { host: "127.6.235.67" port: 42843 } }
I20260928 02:11:41.232728 15214 ts_tablet_manager.cc:1428] T 3239999622d94b1196274986003cfb39 P 476becc74e6c42449c408fc8bc3380b1: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20260928 02:11:41.384547 15263 raft_consensus.cc:482] T 3239999622d94b1196274986003cfb39 P baedbfdc92b946f0a07777ec1f899494 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 02:11:41.384793 15263 raft_consensus.cc:504] T 3239999622d94b1196274986003cfb39 P baedbfdc92b946f0a07777ec1f899494 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7411d0f5db463799e9b2db6669ea48" member_type: VOTER last_known_addr { host: "127.6.235.65" port: 42571 } } peers { permanent_uuid: "476becc74e6c42449c408fc8bc3380b1" member_type: VOTER last_known_addr { host: "127.6.235.66" port: 41335 } } peers { permanent_uuid: "baedbfdc92b946f0a07777ec1f899494" member_type: VOTER last_known_addr { host: "127.6.235.67" port: 42843 } }
I20260928 02:11:41.385742 15263 leader_election.cc:288] T 3239999622d94b1196274986003cfb39 P baedbfdc92b946f0a07777ec1f899494 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6e7411d0f5db463799e9b2db6669ea48 (127.6.235.65:42571), 476becc74e6c42449c408fc8bc3380b1 (127.6.235.66:41335)
I20260928 02:11:41.386386 14853 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3239999622d94b1196274986003cfb39" candidate_uuid: "baedbfdc92b946f0a07777ec1f899494" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6e7411d0f5db463799e9b2db6669ea48" is_pre_election: true
I20260928 02:11:41.386508 14995 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3239999622d94b1196274986003cfb39" candidate_uuid: "baedbfdc92b946f0a07777ec1f899494" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "476becc74e6c42449c408fc8bc3380b1" is_pre_election: true
I20260928 02:11:41.386680 14853 raft_consensus.cc:2454] T 3239999622d94b1196274986003cfb39 P 6e7411d0f5db463799e9b2db6669ea48 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate baedbfdc92b946f0a07777ec1f899494 in term 0.
I20260928 02:11:41.386745 14995 raft_consensus.cc:2454] T 3239999622d94b1196274986003cfb39 P 476becc74e6c42449c408fc8bc3380b1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate baedbfdc92b946f0a07777ec1f899494 in term 0.
I20260928 02:11:41.387274 15067 leader_election.cc:302] T 3239999622d94b1196274986003cfb39 P baedbfdc92b946f0a07777ec1f899494 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 476becc74e6c42449c408fc8bc3380b1, baedbfdc92b946f0a07777ec1f899494; no voters:
I20260928 02:11:41.387660 15263 raft_consensus.cc:2781] T 3239999622d94b1196274986003cfb39 P baedbfdc92b946f0a07777ec1f899494 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 02:11:41.387805 15263 raft_consensus.cc:482] T 3239999622d94b1196274986003cfb39 P baedbfdc92b946f0a07777ec1f899494 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 02:11:41.387908 15263 raft_consensus.cc:3037] T 3239999622d94b1196274986003cfb39 P baedbfdc92b946f0a07777ec1f899494 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:11:41.391448 15263 raft_consensus.cc:504] T 3239999622d94b1196274986003cfb39 P baedbfdc92b946f0a07777ec1f899494 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7411d0f5db463799e9b2db6669ea48" member_type: VOTER last_known_addr { host: "127.6.235.65" port: 42571 } } peers { permanent_uuid: "476becc74e6c42449c408fc8bc3380b1" member_type: VOTER last_known_addr { host: "127.6.235.66" port: 41335 } } peers { permanent_uuid: "baedbfdc92b946f0a07777ec1f899494" member_type: VOTER last_known_addr { host: "127.6.235.67" port: 42843 } }
I20260928 02:11:41.392761 14853 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3239999622d94b1196274986003cfb39" candidate_uuid: "baedbfdc92b946f0a07777ec1f899494" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6e7411d0f5db463799e9b2db6669ea48"
I20260928 02:11:41.393023 14853 raft_consensus.cc:3037] T 3239999622d94b1196274986003cfb39 P 6e7411d0f5db463799e9b2db6669ea48 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:11:41.393152 15263 leader_election.cc:288] T 3239999622d94b1196274986003cfb39 P baedbfdc92b946f0a07777ec1f899494 [CANDIDATE]: Term 1 election: Requested vote from peers 6e7411d0f5db463799e9b2db6669ea48 (127.6.235.65:42571), 476becc74e6c42449c408fc8bc3380b1 (127.6.235.66:41335)
I20260928 02:11:41.393815 14995 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3239999622d94b1196274986003cfb39" candidate_uuid: "baedbfdc92b946f0a07777ec1f899494" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "476becc74e6c42449c408fc8bc3380b1"
I20260928 02:11:41.394061 14995 raft_consensus.cc:3037] T 3239999622d94b1196274986003cfb39 P 476becc74e6c42449c408fc8bc3380b1 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:11:41.396368 14853 raft_consensus.cc:2454] T 3239999622d94b1196274986003cfb39 P 6e7411d0f5db463799e9b2db6669ea48 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate baedbfdc92b946f0a07777ec1f899494 in term 1.
I20260928 02:11:41.396943 15066 leader_election.cc:302] T 3239999622d94b1196274986003cfb39 P baedbfdc92b946f0a07777ec1f899494 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6e7411d0f5db463799e9b2db6669ea48, baedbfdc92b946f0a07777ec1f899494; no voters:
I20260928 02:11:41.397301 15263 raft_consensus.cc:2781] T 3239999622d94b1196274986003cfb39 P baedbfdc92b946f0a07777ec1f899494 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:11:41.397351 14995 raft_consensus.cc:2454] T 3239999622d94b1196274986003cfb39 P 476becc74e6c42449c408fc8bc3380b1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate baedbfdc92b946f0a07777ec1f899494 in term 1.
I20260928 02:11:41.397586 15263 raft_consensus.cc:686] T 3239999622d94b1196274986003cfb39 P baedbfdc92b946f0a07777ec1f899494 [term 1 LEADER]: Becoming Leader. State: Replica: baedbfdc92b946f0a07777ec1f899494, State: Running, Role: LEADER
I20260928 02:11:41.397902 15263 consensus_queue.cc:237] T 3239999622d94b1196274986003cfb39 P baedbfdc92b946f0a07777ec1f899494 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7411d0f5db463799e9b2db6669ea48" member_type: VOTER last_known_addr { host: "127.6.235.65" port: 42571 } } peers { permanent_uuid: "476becc74e6c42449c408fc8bc3380b1" member_type: VOTER last_known_addr { host: "127.6.235.66" port: 41335 } } peers { permanent_uuid: "baedbfdc92b946f0a07777ec1f899494" member_type: VOTER last_known_addr { host: "127.6.235.67" port: 42843 } }
I20260928 02:11:41.400979 14693 catalog_manager.cc:5495] T 3239999622d94b1196274986003cfb39 P baedbfdc92b946f0a07777ec1f899494 reported cstate change: term changed from 0 to 1, leader changed from <none> to baedbfdc92b946f0a07777ec1f899494 (127.6.235.67). New cstate: current_term: 1 leader_uuid: "baedbfdc92b946f0a07777ec1f899494" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e7411d0f5db463799e9b2db6669ea48" member_type: VOTER last_known_addr { host: "127.6.235.65" port: 42571 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "476becc74e6c42449c408fc8bc3380b1" member_type: VOTER last_known_addr { host: "127.6.235.66" port: 41335 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "baedbfdc92b946f0a07777ec1f899494" member_type: VOTER last_known_addr { host: "127.6.235.67" port: 42843 } health_report { overall_health: HEALTHY } } }
I20260928 02:11:41.466369 15291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:11:41.454381 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33404 (local address 127.6.235.67:42843)
0928 02:11:41.455152 (+ 771us) server_negotiation.cc:206] Beginning negotiation
0928 02:11:41.455159 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 02:11:41.459294 (+ 4135us) server_negotiation.cc:407] Connection header received
0928 02:11:41.459396 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:11:41.459401 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:11:41.459495 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:11:41.460103 (+ 608us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 02:11:41.460305 (+ 202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:11:41.461618 (+ 1313us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:11:41.463718 (+ 2100us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:11:41.464566 (+ 848us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:11:41.464735 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:11:41.465360 (+ 625us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 02:11:41.465548 (+ 188us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 02:11:41.465657 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0928 02:11:41.465883 (+ 226us) server_negotiation.cc:299] Negotiation successful
0928 02:11:41.465987 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":575,"thread_start_us":3691,"threads_started":1}
I20260928 02:11:41.470652 14853 raft_consensus.cc:1261] T 3239999622d94b1196274986003cfb39 P 6e7411d0f5db463799e9b2db6669ea48 [term 1 FOLLOWER]: Refusing update from remote peer baedbfdc92b946f0a07777ec1f899494: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 02:11:41.471319 14995 raft_consensus.cc:1261] T 3239999622d94b1196274986003cfb39 P 476becc74e6c42449c408fc8bc3380b1 [term 1 FOLLOWER]: Refusing update from remote peer baedbfdc92b946f0a07777ec1f899494: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 02:11:41.471856 15225 consensus_queue.cc:1035] T 3239999622d94b1196274986003cfb39 P baedbfdc92b946f0a07777ec1f899494 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e7411d0f5db463799e9b2db6669ea48" member_type: VOTER last_known_addr { host: "127.6.235.65" port: 42571 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 02:11:41.472188 15263 consensus_queue.cc:1035] T 3239999622d94b1196274986003cfb39 P baedbfdc92b946f0a07777ec1f899494 [LEADER]: Connected to new peer: Peer: permanent_uuid: "476becc74e6c42449c408fc8bc3380b1" member_type: VOTER last_known_addr { host: "127.6.235.66" port: 41335 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 02:11:41.496461 14693 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55466:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 02:11:41.496796 14693 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55466:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 02:11:41.501170 14693 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 03972cb2d8744214a6a4570fb5531fe0: Sending DeleteTablet for 3 replicas of tablet 3239999622d94b1196274986003cfb39
I20260928 02:11:41.501947 14974 tablet_service.cc:1514] Processing DeleteTablet for tablet 3239999622d94b1196274986003cfb39 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:11:41 UTC) from {username='oryx', principal='oryx/127.6.235.126@KRBTEST.COM'} at 127.0.0.1:34518
I20260928 02:11:41.502585 15240 tablet_replica.cc:330] T 3239999622d94b1196274986003cfb39 P 476becc74e6c42449c408fc8bc3380b1: stopping tablet replica
I20260928 02:11:41.502794 15240 raft_consensus.cc:2229] T 3239999622d94b1196274986003cfb39 P 476becc74e6c42449c408fc8bc3380b1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 02:11:41.503079 15240 raft_consensus.cc:2258] T 3239999622d94b1196274986003cfb39 P 476becc74e6c42449c408fc8bc3380b1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 02:11:41.503456 14832 tablet_service.cc:1514] Processing DeleteTablet for tablet 3239999622d94b1196274986003cfb39 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:11:41 UTC) from {username='oryx', principal='oryx/127.6.235.126@KRBTEST.COM'} at 127.0.0.1:49468
I20260928 02:11:41.504201 15239 tablet_replica.cc:330] T 3239999622d94b1196274986003cfb39 P 6e7411d0f5db463799e9b2db6669ea48: stopping tablet replica
I20260928 02:11:41.504388 15239 raft_consensus.cc:2229] T 3239999622d94b1196274986003cfb39 P 6e7411d0f5db463799e9b2db6669ea48 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 02:11:41.504644 15239 raft_consensus.cc:2258] T 3239999622d94b1196274986003cfb39 P 6e7411d0f5db463799e9b2db6669ea48 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 02:11:41.505422 15117 tablet_service.cc:1514] Processing DeleteTablet for tablet 3239999622d94b1196274986003cfb39 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:11:41 UTC) from {username='oryx', principal='oryx/127.6.235.126@KRBTEST.COM'} at 127.0.0.1:39054
I20260928 02:11:41.505750 15238 tablet_replica.cc:330] T 3239999622d94b1196274986003cfb39 P baedbfdc92b946f0a07777ec1f899494: stopping tablet replica
I20260928 02:11:41.505927 15238 raft_consensus.cc:2229] T 3239999622d94b1196274986003cfb39 P baedbfdc92b946f0a07777ec1f899494 [term 1 LEADER]: Raft consensus shutting down.
I20260928 02:11:41.506292 15238 raft_consensus.cc:2258] T 3239999622d94b1196274986003cfb39 P baedbfdc92b946f0a07777ec1f899494 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 02:11:41.506891 15116 tablet_service.cc:1514] Processing DeleteTablet for tablet 3239999622d94b1196274986003cfb39 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:11:41 UTC) from {username='oryx', principal='oryx/127.6.235.126@KRBTEST.COM'} at 127.0.0.1:39054
I20260928 02:11:41.506990 15239 ts_tablet_manager.cc:1905] T 3239999622d94b1196274986003cfb39 P 6e7411d0f5db463799e9b2db6669ea48: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 02:11:41.509222 15238 ts_tablet_manager.cc:1905] T 3239999622d94b1196274986003cfb39 P baedbfdc92b946f0a07777ec1f899494: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 02:11:41.513867 14693 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55466:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 02:11:41.514605 14693 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 02:11:41.519218 15238 ts_tablet_manager.cc:1918] T 3239999622d94b1196274986003cfb39 P baedbfdc92b946f0a07777ec1f899494: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 02:11:41.519522 15238 log.cc:1196] T 3239999622d94b1196274986003cfb39 P baedbfdc92b946f0a07777ec1f899494: Deleting WAL directory at /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790561432251112-7085-0/minicluster-data/ts-2/wal/wals/3239999622d94b1196274986003cfb39
I20260928 02:11:41.520219 15238 ts_tablet_manager.cc:1939] T 3239999622d94b1196274986003cfb39 P baedbfdc92b946f0a07777ec1f899494: Deleting consensus metadata
I20260928 02:11:41.521559 14681 catalog_manager.cc:4841] TS baedbfdc92b946f0a07777ec1f899494 (127.6.235.67:42843): tablet 3239999622d94b1196274986003cfb39 (table test-table [id=ebade711c2ea4302a98c82d42686c0e0]) successfully deleted
I20260928 02:11:41.522936 14974 tablet_service.cc:1467] Processing CreateTablet for tablet 25bff2e1d9944b059856af4391733c3a (DEFAULT_TABLE table=test-table [id=f5113e3ba6304d1f94d5dd35d4e76985]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:11:41.523386 14974 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 25bff2e1d9944b059856af4391733c3a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:11:41.523406 14832 tablet_service.cc:1467] Processing CreateTablet for tablet 25bff2e1d9944b059856af4391733c3a (DEFAULT_TABLE table=test-table [id=f5113e3ba6304d1f94d5dd35d4e76985]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:11:41.523874 14832 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 25bff2e1d9944b059856af4391733c3a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:11:41.529969 15116 tablet_service.cc:1467] Processing CreateTablet for tablet 25bff2e1d9944b059856af4391733c3a (DEFAULT_TABLE table=test-table [id=f5113e3ba6304d1f94d5dd35d4e76985]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:11:41.530580 15116 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 25bff2e1d9944b059856af4391733c3a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:11:41.538360 15239 ts_tablet_manager.cc:1918] T 3239999622d94b1196274986003cfb39 P 6e7411d0f5db463799e9b2db6669ea48: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 02:11:41.538573 15239 log.cc:1196] T 3239999622d94b1196274986003cfb39 P 6e7411d0f5db463799e9b2db6669ea48: Deleting WAL directory at /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790561432251112-7085-0/minicluster-data/ts-0/wal/wals/3239999622d94b1196274986003cfb39
I20260928 02:11:41.539099 15239 ts_tablet_manager.cc:1939] T 3239999622d94b1196274986003cfb39 P 6e7411d0f5db463799e9b2db6669ea48: Deleting consensus metadata
I20260928 02:11:41.540369 14678 catalog_manager.cc:4841] TS 6e7411d0f5db463799e9b2db6669ea48 (127.6.235.65:42571): tablet 3239999622d94b1196274986003cfb39 (table test-table [id=ebade711c2ea4302a98c82d42686c0e0]) successfully deleted
I20260928 02:11:41.541764 15216 tablet_bootstrap.cc:492] T 25bff2e1d9944b059856af4391733c3a P 6e7411d0f5db463799e9b2db6669ea48: Bootstrap starting.
I20260928 02:11:41.542215 15214 tablet_bootstrap.cc:492] T 25bff2e1d9944b059856af4391733c3a P 476becc74e6c42449c408fc8bc3380b1: Bootstrap starting.
I20260928 02:11:41.543763 15216 tablet_bootstrap.cc:654] T 25bff2e1d9944b059856af4391733c3a P 6e7411d0f5db463799e9b2db6669ea48: Neither blocks nor log segments found. Creating new log.
I20260928 02:11:41.543941 15214 tablet_bootstrap.cc:654] T 25bff2e1d9944b059856af4391733c3a P 476becc74e6c42449c408fc8bc3380b1: Neither blocks nor log segments found. Creating new log.
I20260928 02:11:41.549183 15240 ts_tablet_manager.cc:1905] T 3239999622d94b1196274986003cfb39 P 476becc74e6c42449c408fc8bc3380b1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 02:11:41.553570 15216 tablet_bootstrap.cc:492] T 25bff2e1d9944b059856af4391733c3a P 6e7411d0f5db463799e9b2db6669ea48: No bootstrap required, opened a new log
I20260928 02:11:41.553790 15216 ts_tablet_manager.cc:1397] T 25bff2e1d9944b059856af4391733c3a P 6e7411d0f5db463799e9b2db6669ea48: Time spent bootstrapping tablet: real 0.012s user 0.005s sys 0.004s
I20260928 02:11:41.554605 15216 raft_consensus.cc:348] T 25bff2e1d9944b059856af4391733c3a P 6e7411d0f5db463799e9b2db6669ea48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "476becc74e6c42449c408fc8bc3380b1" member_type: VOTER last_known_addr { host: "127.6.235.66" port: 41335 } } peers { permanent_uuid: "6e7411d0f5db463799e9b2db6669ea48" member_type: VOTER last_known_addr { host: "127.6.235.65" port: 42571 } } peers { permanent_uuid: "baedbfdc92b946f0a07777ec1f899494" member_type: VOTER last_known_addr { host: "127.6.235.67" port: 42843 } }
I20260928 02:11:41.554939 15216 raft_consensus.cc:374] T 25bff2e1d9944b059856af4391733c3a P 6e7411d0f5db463799e9b2db6669ea48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:11:41.555075 15216 raft_consensus.cc:729] T 25bff2e1d9944b059856af4391733c3a P 6e7411d0f5db463799e9b2db6669ea48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6e7411d0f5db463799e9b2db6669ea48, State: Initialized, Role: FOLLOWER
I20260928 02:11:41.555457 15216 consensus_queue.cc:260] T 25bff2e1d9944b059856af4391733c3a P 6e7411d0f5db463799e9b2db6669ea48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "476becc74e6c42449c408fc8bc3380b1" member_type: VOTER last_known_addr { host: "127.6.235.66" port: 41335 } } peers { permanent_uuid: "6e7411d0f5db463799e9b2db6669ea48" member_type: VOTER last_known_addr { host: "127.6.235.65" port: 42571 } } peers { permanent_uuid: "baedbfdc92b946f0a07777ec1f899494" member_type: VOTER last_known_addr { host: "127.6.235.67" port: 42843 } }
I20260928 02:11:41.557364 15216 ts_tablet_manager.cc:1428] T 25bff2e1d9944b059856af4391733c3a P 6e7411d0f5db463799e9b2db6669ea48: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260928 02:11:41.562392 15215 tablet_bootstrap.cc:492] T 25bff2e1d9944b059856af4391733c3a P baedbfdc92b946f0a07777ec1f899494: Bootstrap starting.
I20260928 02:11:41.560985 15240 ts_tablet_manager.cc:1918] T 3239999622d94b1196274986003cfb39 P 476becc74e6c42449c408fc8bc3380b1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 02:11:41.563158 15240 log.cc:1196] T 3239999622d94b1196274986003cfb39 P 476becc74e6c42449c408fc8bc3380b1: Deleting WAL directory at /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790561432251112-7085-0/minicluster-data/ts-1/wal/wals/3239999622d94b1196274986003cfb39
I20260928 02:11:41.563572 15240 ts_tablet_manager.cc:1939] T 3239999622d94b1196274986003cfb39 P 476becc74e6c42449c408fc8bc3380b1: Deleting consensus metadata
I20260928 02:11:41.564301 15215 tablet_bootstrap.cc:654] T 25bff2e1d9944b059856af4391733c3a P baedbfdc92b946f0a07777ec1f899494: Neither blocks nor log segments found. Creating new log.
I20260928 02:11:41.564517 14679 catalog_manager.cc:4841] TS 476becc74e6c42449c408fc8bc3380b1 (127.6.235.66:41335): tablet 3239999622d94b1196274986003cfb39 (table test-table [id=ebade711c2ea4302a98c82d42686c0e0]) successfully deleted
I20260928 02:11:41.565801 15214 tablet_bootstrap.cc:492] T 25bff2e1d9944b059856af4391733c3a P 476becc74e6c42449c408fc8bc3380b1: No bootstrap required, opened a new log
I20260928 02:11:41.566005 15214 ts_tablet_manager.cc:1397] T 25bff2e1d9944b059856af4391733c3a P 476becc74e6c42449c408fc8bc3380b1: Time spent bootstrapping tablet: real 0.024s user 0.003s sys 0.002s
I20260928 02:11:41.566773 15214 raft_consensus.cc:348] T 25bff2e1d9944b059856af4391733c3a P 476becc74e6c42449c408fc8bc3380b1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "476becc74e6c42449c408fc8bc3380b1" member_type: VOTER last_known_addr { host: "127.6.235.66" port: 41335 } } peers { permanent_uuid: "6e7411d0f5db463799e9b2db6669ea48" member_type: VOTER last_known_addr { host: "127.6.235.65" port: 42571 } } peers { permanent_uuid: "baedbfdc92b946f0a07777ec1f899494" member_type: VOTER last_known_addr { host: "127.6.235.67" port: 42843 } }
I20260928 02:11:41.567812 15214 raft_consensus.cc:374] T 25bff2e1d9944b059856af4391733c3a P 476becc74e6c42449c408fc8bc3380b1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:11:41.567922 15214 raft_consensus.cc:729] T 25bff2e1d9944b059856af4391733c3a P 476becc74e6c42449c408fc8bc3380b1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 476becc74e6c42449c408fc8bc3380b1, State: Initialized, Role: FOLLOWER
I20260928 02:11:41.568188 15214 consensus_queue.cc:260] T 25bff2e1d9944b059856af4391733c3a P 476becc74e6c42449c408fc8bc3380b1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "476becc74e6c42449c408fc8bc3380b1" member_type: VOTER last_known_addr { host: "127.6.235.66" port: 41335 } } peers { permanent_uuid: "6e7411d0f5db463799e9b2db6669ea48" member_type: VOTER last_known_addr { host: "127.6.235.65" port: 42571 } } peers { permanent_uuid: "baedbfdc92b946f0a07777ec1f899494" member_type: VOTER last_known_addr { host: "127.6.235.67" port: 42843 } }
I20260928 02:11:41.570947 15214 ts_tablet_manager.cc:1428] T 25bff2e1d9944b059856af4391733c3a P 476becc74e6c42449c408fc8bc3380b1: Time spent starting tablet: real 0.005s user 0.001s sys 0.001s
W20260928 02:11:41.574218 14681 catalog_manager.cc:4808] TS baedbfdc92b946f0a07777ec1f899494 (127.6.235.67:42843): delete failed for tablet 3239999622d94b1196274986003cfb39 because the tablet was not found. No further retry: Not found: Tablet not found: 3239999622d94b1196274986003cfb39
I20260928 02:11:41.576306 15215 tablet_bootstrap.cc:492] T 25bff2e1d9944b059856af4391733c3a P baedbfdc92b946f0a07777ec1f899494: No bootstrap required, opened a new log
I20260928 02:11:41.576514 15215 ts_tablet_manager.cc:1397] T 25bff2e1d9944b059856af4391733c3a P baedbfdc92b946f0a07777ec1f899494: Time spent bootstrapping tablet: real 0.014s user 0.004s sys 0.007s
I20260928 02:11:41.577515 15215 raft_consensus.cc:348] T 25bff2e1d9944b059856af4391733c3a P baedbfdc92b946f0a07777ec1f899494 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "476becc74e6c42449c408fc8bc3380b1" member_type: VOTER last_known_addr { host: "127.6.235.66" port: 41335 } } peers { permanent_uuid: "6e7411d0f5db463799e9b2db6669ea48" member_type: VOTER last_known_addr { host: "127.6.235.65" port: 42571 } } peers { permanent_uuid: "baedbfdc92b946f0a07777ec1f899494" member_type: VOTER last_known_addr { host: "127.6.235.67" port: 42843 } }
I20260928 02:11:41.577886 15215 raft_consensus.cc:374] T 25bff2e1d9944b059856af4391733c3a P baedbfdc92b946f0a07777ec1f899494 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:11:41.577981 15215 raft_consensus.cc:729] T 25bff2e1d9944b059856af4391733c3a P baedbfdc92b946f0a07777ec1f899494 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: baedbfdc92b946f0a07777ec1f899494, State: Initialized, Role: FOLLOWER
I20260928 02:11:41.578239 15215 consensus_queue.cc:260] T 25bff2e1d9944b059856af4391733c3a P baedbfdc92b946f0a07777ec1f899494 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "476becc74e6c42449c408fc8bc3380b1" member_type: VOTER last_known_addr { host: "127.6.235.66" port: 41335 } } peers { permanent_uuid: "6e7411d0f5db463799e9b2db6669ea48" member_type: VOTER last_known_addr { host: "127.6.235.65" port: 42571 } } peers { permanent_uuid: "baedbfdc92b946f0a07777ec1f899494" member_type: VOTER last_known_addr { host: "127.6.235.67" port: 42843 } }
I20260928 02:11:41.578855 15215 ts_tablet_manager.cc:1428] T 25bff2e1d9944b059856af4391733c3a P baedbfdc92b946f0a07777ec1f899494: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260928 02:11:41.690059 15246 raft_consensus.cc:482] T 25bff2e1d9944b059856af4391733c3a P 6e7411d0f5db463799e9b2db6669ea48 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 02:11:41.690299 15246 raft_consensus.cc:504] T 25bff2e1d9944b059856af4391733c3a P 6e7411d0f5db463799e9b2db6669ea48 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "476becc74e6c42449c408fc8bc3380b1" member_type: VOTER last_known_addr { host: "127.6.235.66" port: 41335 } } peers { permanent_uuid: "6e7411d0f5db463799e9b2db6669ea48" member_type: VOTER last_known_addr { host: "127.6.235.65" port: 42571 } } peers { permanent_uuid: "baedbfdc92b946f0a07777ec1f899494" member_type: VOTER last_known_addr { host: "127.6.235.67" port: 42843 } }
I20260928 02:11:41.691004 15246 leader_election.cc:288] T 25bff2e1d9944b059856af4391733c3a P 6e7411d0f5db463799e9b2db6669ea48 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 476becc74e6c42449c408fc8bc3380b1 (127.6.235.66:41335), baedbfdc92b946f0a07777ec1f899494 (127.6.235.67:42843)
I20260928 02:11:41.691622 14995 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "25bff2e1d9944b059856af4391733c3a" candidate_uuid: "6e7411d0f5db463799e9b2db6669ea48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "476becc74e6c42449c408fc8bc3380b1" is_pre_election: true
I20260928 02:11:41.691700 15141 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "25bff2e1d9944b059856af4391733c3a" candidate_uuid: "6e7411d0f5db463799e9b2db6669ea48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "baedbfdc92b946f0a07777ec1f899494" is_pre_election: true
I20260928 02:11:41.691877 14995 raft_consensus.cc:2454] T 25bff2e1d9944b059856af4391733c3a P 476becc74e6c42449c408fc8bc3380b1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e7411d0f5db463799e9b2db6669ea48 in term 0.
I20260928 02:11:41.692013 15141 raft_consensus.cc:2454] T 25bff2e1d9944b059856af4391733c3a P baedbfdc92b946f0a07777ec1f899494 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e7411d0f5db463799e9b2db6669ea48 in term 0.
I20260928 02:11:41.692381 14783 leader_election.cc:302] T 25bff2e1d9944b059856af4391733c3a P 6e7411d0f5db463799e9b2db6669ea48 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 476becc74e6c42449c408fc8bc3380b1, 6e7411d0f5db463799e9b2db6669ea48; no voters:
I20260928 02:11:41.692708 15246 raft_consensus.cc:2781] T 25bff2e1d9944b059856af4391733c3a P 6e7411d0f5db463799e9b2db6669ea48 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 02:11:41.692816 15246 raft_consensus.cc:482] T 25bff2e1d9944b059856af4391733c3a P 6e7411d0f5db463799e9b2db6669ea48 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 02:11:41.692996 15246 raft_consensus.cc:3037] T 25bff2e1d9944b059856af4391733c3a P 6e7411d0f5db463799e9b2db6669ea48 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:11:41.695210 15246 raft_consensus.cc:504] T 25bff2e1d9944b059856af4391733c3a P 6e7411d0f5db463799e9b2db6669ea48 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "476becc74e6c42449c408fc8bc3380b1" member_type: VOTER last_known_addr { host: "127.6.235.66" port: 41335 } } peers { permanent_uuid: "6e7411d0f5db463799e9b2db6669ea48" member_type: VOTER last_known_addr { host: "127.6.235.65" port: 42571 } } peers { permanent_uuid: "baedbfdc92b946f0a07777ec1f899494" member_type: VOTER last_known_addr { host: "127.6.235.67" port: 42843 } }
I20260928 02:11:41.696029 15246 leader_election.cc:288] T 25bff2e1d9944b059856af4391733c3a P 6e7411d0f5db463799e9b2db6669ea48 [CANDIDATE]: Term 1 election: Requested vote from peers 476becc74e6c42449c408fc8bc3380b1 (127.6.235.66:41335), baedbfdc92b946f0a07777ec1f899494 (127.6.235.67:42843)
I20260928 02:11:41.696614 15141 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "25bff2e1d9944b059856af4391733c3a" candidate_uuid: "6e7411d0f5db463799e9b2db6669ea48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "baedbfdc92b946f0a07777ec1f899494"
I20260928 02:11:41.696589 14995 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "25bff2e1d9944b059856af4391733c3a" candidate_uuid: "6e7411d0f5db463799e9b2db6669ea48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "476becc74e6c42449c408fc8bc3380b1"
I20260928 02:11:41.696835 15141 raft_consensus.cc:3037] T 25bff2e1d9944b059856af4391733c3a P baedbfdc92b946f0a07777ec1f899494 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:11:41.696867 14995 raft_consensus.cc:3037] T 25bff2e1d9944b059856af4391733c3a P 476becc74e6c42449c408fc8bc3380b1 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:11:41.699083 14995 raft_consensus.cc:2454] T 25bff2e1d9944b059856af4391733c3a P 476becc74e6c42449c408fc8bc3380b1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6e7411d0f5db463799e9b2db6669ea48 in term 1.
I20260928 02:11:41.699105 15141 raft_consensus.cc:2454] T 25bff2e1d9944b059856af4391733c3a P baedbfdc92b946f0a07777ec1f899494 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6e7411d0f5db463799e9b2db6669ea48 in term 1.
I20260928 02:11:41.699586 14783 leader_election.cc:302] T 25bff2e1d9944b059856af4391733c3a P 6e7411d0f5db463799e9b2db6669ea48 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 476becc74e6c42449c408fc8bc3380b1, 6e7411d0f5db463799e9b2db6669ea48; no voters:
I20260928 02:11:41.699900 15246 raft_consensus.cc:2781] T 25bff2e1d9944b059856af4391733c3a P 6e7411d0f5db463799e9b2db6669ea48 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:11:41.700325 15246 raft_consensus.cc:686] T 25bff2e1d9944b059856af4391733c3a P 6e7411d0f5db463799e9b2db6669ea48 [term 1 LEADER]: Becoming Leader. State: Replica: 6e7411d0f5db463799e9b2db6669ea48, State: Running, Role: LEADER
I20260928 02:11:41.700590 15246 consensus_queue.cc:237] T 25bff2e1d9944b059856af4391733c3a P 6e7411d0f5db463799e9b2db6669ea48 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "476becc74e6c42449c408fc8bc3380b1" member_type: VOTER last_known_addr { host: "127.6.235.66" port: 41335 } } peers { permanent_uuid: "6e7411d0f5db463799e9b2db6669ea48" member_type: VOTER last_known_addr { host: "127.6.235.65" port: 42571 } } peers { permanent_uuid: "baedbfdc92b946f0a07777ec1f899494" member_type: VOTER last_known_addr { host: "127.6.235.67" port: 42843 } }
I20260928 02:11:41.703266 14693 catalog_manager.cc:5495] T 25bff2e1d9944b059856af4391733c3a P 6e7411d0f5db463799e9b2db6669ea48 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6e7411d0f5db463799e9b2db6669ea48 (127.6.235.65). New cstate: current_term: 1 leader_uuid: "6e7411d0f5db463799e9b2db6669ea48" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "476becc74e6c42449c408fc8bc3380b1" member_type: VOTER last_known_addr { host: "127.6.235.66" port: 41335 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6e7411d0f5db463799e9b2db6669ea48" member_type: VOTER last_known_addr { host: "127.6.235.65" port: 42571 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "baedbfdc92b946f0a07777ec1f899494" member_type: VOTER last_known_addr { host: "127.6.235.67" port: 42843 } health_report { overall_health: UNKNOWN } } }
I20260928 02:11:41.725792 15299 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:11:41.720776 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58610 (local address 127.6.235.65:42571)
0928 02:11:41.721325 (+ 549us) server_negotiation.cc:206] Beginning negotiation
0928 02:11:41.721329 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:11:41.721345 (+ 16us) server_negotiation.cc:407] Connection header received
0928 02:11:41.721397 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:11:41.721400 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:11:41.721458 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:11:41.721539 (+ 81us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 02:11:41.721937 (+ 398us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:11:41.722698 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:11:41.723992 (+ 1294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:11:41.724400 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:11:41.724488 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:11:41.724993 (+ 505us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 02:11:41.725135 (+ 142us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 02:11:41.725207 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0928 02:11:41.725352 (+ 145us) server_negotiation.cc:299] Negotiation successful
0928 02:11:41.725412 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":438,"thread_start_us":268,"threads_started":1}
I20260928 02:11:41.735545 14995 raft_consensus.cc:1261] T 25bff2e1d9944b059856af4391733c3a P 476becc74e6c42449c408fc8bc3380b1 [term 1 FOLLOWER]: Refusing update from remote peer 6e7411d0f5db463799e9b2db6669ea48: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 02:11:41.735675 15141 raft_consensus.cc:1261] T 25bff2e1d9944b059856af4391733c3a P baedbfdc92b946f0a07777ec1f899494 [term 1 FOLLOWER]: Refusing update from remote peer 6e7411d0f5db463799e9b2db6669ea48: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 02:11:41.736250 15297 consensus_queue.cc:1035] T 25bff2e1d9944b059856af4391733c3a P 6e7411d0f5db463799e9b2db6669ea48 [LEADER]: Connected to new peer: Peer: permanent_uuid: "476becc74e6c42449c408fc8bc3380b1" member_type: VOTER last_known_addr { host: "127.6.235.66" port: 41335 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 02:11:41.736605 15246 consensus_queue.cc:1035] T 25bff2e1d9944b059856af4391733c3a P 6e7411d0f5db463799e9b2db6669ea48 [LEADER]: Connected to new peer: Peer: permanent_uuid: "baedbfdc92b946f0a07777ec1f899494" member_type: VOTER last_known_addr { host: "127.6.235.67" port: 42843 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 02:11:41.787132 14693 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55466:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 02:11:41.787433 14693 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55466:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 02:11:41.790720 14693 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 03972cb2d8744214a6a4570fb5531fe0: Sending DeleteTablet for 3 replicas of tablet 25bff2e1d9944b059856af4391733c3a
I20260928 02:11:41.791690 14974 tablet_service.cc:1514] Processing DeleteTablet for tablet 25bff2e1d9944b059856af4391733c3a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:11:41 UTC) from {username='oryx', principal='oryx/127.6.235.126@KRBTEST.COM'} at 127.0.0.1:34518
I20260928 02:11:41.791714 15116 tablet_service.cc:1514] Processing DeleteTablet for tablet 25bff2e1d9944b059856af4391733c3a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:11:41 UTC) from {username='oryx', principal='oryx/127.6.235.126@KRBTEST.COM'} at 127.0.0.1:39054
I20260928 02:11:41.791747 14832 tablet_service.cc:1514] Processing DeleteTablet for tablet 25bff2e1d9944b059856af4391733c3a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:11:41 UTC) from {username='oryx', principal='oryx/127.6.235.126@KRBTEST.COM'} at 127.0.0.1:49468
I20260928 02:11:41.792004 15240 tablet_replica.cc:330] T 25bff2e1d9944b059856af4391733c3a P 476becc74e6c42449c408fc8bc3380b1: stopping tablet replica
I20260928 02:11:41.792012 15239 tablet_replica.cc:330] T 25bff2e1d9944b059856af4391733c3a P 6e7411d0f5db463799e9b2db6669ea48: stopping tablet replica
I20260928 02:11:41.792203 15240 raft_consensus.cc:2229] T 25bff2e1d9944b059856af4391733c3a P 476becc74e6c42449c408fc8bc3380b1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 02:11:41.792263 15239 raft_consensus.cc:2229] T 25bff2e1d9944b059856af4391733c3a P 6e7411d0f5db463799e9b2db6669ea48 [term 1 LEADER]: Raft consensus shutting down.
I20260928 02:11:41.792407 15240 raft_consensus.cc:2258] T 25bff2e1d9944b059856af4391733c3a P 476becc74e6c42449c408fc8bc3380b1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 02:11:41.792672 15239 raft_consensus.cc:2258] T 25bff2e1d9944b059856af4391733c3a P 6e7411d0f5db463799e9b2db6669ea48 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 02:11:41.793193 7085 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskTlaUuc/build/asan/bin/kudu pid 14766
I20260928 02:11:41.794435 15240 ts_tablet_manager.cc:1905] T 25bff2e1d9944b059856af4391733c3a P 476becc74e6c42449c408fc8bc3380b1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 02:11:41.794560 15239 ts_tablet_manager.cc:1905] T 25bff2e1d9944b059856af4391733c3a P 6e7411d0f5db463799e9b2db6669ea48: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 02:11:41.798466 15238 tablet_replica.cc:330] T 25bff2e1d9944b059856af4391733c3a P baedbfdc92b946f0a07777ec1f899494: stopping tablet replica
I20260928 02:11:41.798696 15238 raft_consensus.cc:2229] T 25bff2e1d9944b059856af4391733c3a P baedbfdc92b946f0a07777ec1f899494 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 02:11:41.798913 15238 raft_consensus.cc:2258] T 25bff2e1d9944b059856af4391733c3a P baedbfdc92b946f0a07777ec1f899494 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 02:11:41.799927 15240 ts_tablet_manager.cc:1918] T 25bff2e1d9944b059856af4391733c3a P 476becc74e6c42449c408fc8bc3380b1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 02:11:41.800033 15238 ts_tablet_manager.cc:1905] T 25bff2e1d9944b059856af4391733c3a P baedbfdc92b946f0a07777ec1f899494: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 02:11:41.800086 15240 log.cc:1196] T 25bff2e1d9944b059856af4391733c3a P 476becc74e6c42449c408fc8bc3380b1: Deleting WAL directory at /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790561432251112-7085-0/minicluster-data/ts-1/wal/wals/25bff2e1d9944b059856af4391733c3a
I20260928 02:11:41.800485 15240 ts_tablet_manager.cc:1939] T 25bff2e1d9944b059856af4391733c3a P 476becc74e6c42449c408fc8bc3380b1: Deleting consensus metadata
I20260928 02:11:41.801963 14679 catalog_manager.cc:4841] TS 476becc74e6c42449c408fc8bc3380b1 (127.6.235.66:41335): tablet 25bff2e1d9944b059856af4391733c3a (table test-table [id=f5113e3ba6304d1f94d5dd35d4e76985]) successfully deleted
I20260928 02:11:41.803354 15239 ts_tablet_manager.cc:1918] T 25bff2e1d9944b059856af4391733c3a P 6e7411d0f5db463799e9b2db6669ea48: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 02:11:41.803552 15239 log.cc:1196] T 25bff2e1d9944b059856af4391733c3a P 6e7411d0f5db463799e9b2db6669ea48: Deleting WAL directory at /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790561432251112-7085-0/minicluster-data/ts-0/wal/wals/25bff2e1d9944b059856af4391733c3a
I20260928 02:11:41.803956 15239 ts_tablet_manager.cc:1939] T 25bff2e1d9944b059856af4391733c3a P 6e7411d0f5db463799e9b2db6669ea48: Deleting consensus metadata
Sep 28 02:11:41 dist-test-slave-86gv krb5kdc[14649](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561501, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.6.235.65@KRBTEST.COM
I20260928 02:11:41.805171 14678 catalog_manager.cc:4841] TS 6e7411d0f5db463799e9b2db6669ea48 (127.6.235.65:42571): tablet 25bff2e1d9944b059856af4391733c3a (table test-table [id=f5113e3ba6304d1f94d5dd35d4e76985]) successfully deleted
I20260928 02:11:41.806005 15238 ts_tablet_manager.cc:1918] T 25bff2e1d9944b059856af4391733c3a P baedbfdc92b946f0a07777ec1f899494: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 02:11:41.806167 15238 log.cc:1196] T 25bff2e1d9944b059856af4391733c3a P baedbfdc92b946f0a07777ec1f899494: Deleting WAL directory at /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790561432251112-7085-0/minicluster-data/ts-2/wal/wals/25bff2e1d9944b059856af4391733c3a
I20260928 02:11:41.806540 15238 ts_tablet_manager.cc:1939] T 25bff2e1d9944b059856af4391733c3a P baedbfdc92b946f0a07777ec1f899494: Deleting consensus metadata
I20260928 02:11:41.807719 14681 catalog_manager.cc:4841] TS baedbfdc92b946f0a07777ec1f899494 (127.6.235.67:42843): tablet 25bff2e1d9944b059856af4391733c3a (table test-table [id=f5113e3ba6304d1f94d5dd35d4e76985]) successfully deleted
I20260928 02:11:41.812600 15299 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:11:41.794481 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58614 (local address 127.6.235.65:42571)
0928 02:11:41.794662 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0928 02:11:41.794668 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 02:11:41.794688 (+ 20us) server_negotiation.cc:407] Connection header received
0928 02:11:41.794867 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:11:41.794873 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:11:41.794952 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:11:41.795029 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:11:41.796794 (+ 1765us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:11:41.797969 (+ 1175us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:11:41.799921 (+ 1952us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:11:41.800606 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:11:41.800733 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:11:41.805893 (+ 5160us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:11:41.805916 (+ 23us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:11:41.805933 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:11:41.805975 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:11:41.809726 (+ 3751us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:11:41.810552 (+ 826us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:11:41.810561 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:11:41.810564 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:11:41.810718 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:11:41.811162 (+ 444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:11:41.811166 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:11:41.811168 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:11:41.811545 (+ 377us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:11:41.811798 (+ 253us) server_negotiation.cc:1035] Waiting for connection context
0928 02:11:41.812193 (+ 395us) server_negotiation.cc:299] Negotiation successful
0928 02:11:41.812369 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260928 02:11:41.813387 14898 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 02:11:41.941807 7085 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTlaUuc/build/asan/bin/kudu with pid 14766
I20260928 02:11:41.967079 7085 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskTlaUuc/build/asan/bin/kudu pid 14907
Sep 28 02:11:41 dist-test-slave-86gv krb5kdc[14649](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561501, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.6.235.66@KRBTEST.COM
I20260928 02:11:41.984160 15308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:11:41.968000 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57746 (local address 127.6.235.66:41335)
0928 02:11:41.968556 (+ 556us) server_negotiation.cc:206] Beginning negotiation
0928 02:11:41.968560 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:11:41.968574 (+ 14us) server_negotiation.cc:407] Connection header received
0928 02:11:41.968655 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:11:41.968658 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:11:41.968716 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:11:41.968845 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:11:41.970075 (+ 1230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:11:41.970840 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:11:41.972102 (+ 1262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:11:41.972530 (+ 428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:11:41.972712 (+ 182us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:11:41.977666 (+ 4954us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:11:41.977684 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:11:41.977699 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:11:41.977751 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:11:41.980880 (+ 3129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:11:41.981792 (+ 912us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:11:41.981801 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:11:41.981803 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:11:41.982034 (+ 231us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:11:41.982452 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:11:41.982457 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:11:41.982459 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:11:41.982867 (+ 408us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:11:41.983167 (+ 300us) server_negotiation.cc:1035] Waiting for connection context
0928 02:11:41.983684 (+ 517us) server_negotiation.cc:299] Negotiation successful
0928 02:11:41.983866 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":438,"thread_start_us":282,"threads_started":1}
I20260928 02:11:41.985203 15041 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 02:11:42.086529 7085 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTlaUuc/build/asan/bin/kudu with pid 14907
I20260928 02:11:42.105465 7085 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskTlaUuc/build/asan/bin/kudu pid 15050
Sep 28 02:11:42 dist-test-slave-86gv krb5kdc[14649](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561501, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.6.235.67@KRBTEST.COM
I20260928 02:11:42.123699 15310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:11:42.106282 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33418 (local address 127.6.235.67:42843)
0928 02:11:42.106949 (+ 667us) server_negotiation.cc:206] Beginning negotiation
0928 02:11:42.106959 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 02:11:42.106979 (+ 20us) server_negotiation.cc:407] Connection header received
0928 02:11:42.107099 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:11:42.107103 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:11:42.107161 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:11:42.107259 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:11:42.108641 (+ 1382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:11:42.109517 (+ 876us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:11:42.111038 (+ 1521us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:11:42.111445 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:11:42.111562 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:11:42.116844 (+ 5282us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:11:42.116864 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:11:42.116882 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:11:42.117004 (+ 122us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:11:42.120192 (+ 3188us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:11:42.121161 (+ 969us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:11:42.121175 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:11:42.121182 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:11:42.121352 (+ 170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:11:42.121808 (+ 456us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:11:42.121812 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:11:42.121814 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:11:42.122290 (+ 476us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:11:42.122552 (+ 262us) server_negotiation.cc:1035] Waiting for connection context
0928 02:11:42.123105 (+ 553us) server_negotiation.cc:299] Negotiation successful
0928 02:11:42.123382 (+ 277us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":571,"thread_start_us":479,"threads_started":1}
I20260928 02:11:42.124445 15182 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 02:11:42.274135 15313 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:11:42.272794 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.235.66:41335 (local address 127.6.235.67:42661)
0928 02:11:42.273679 (+ 885us) negotiation.cc:107] Waiting for socket to connect
0928 02:11:42.273864 (+ 185us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.235.66:41335: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":748,"threads_started":1}
I20260928 02:11:42.274588 15313 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:11:42.273152 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.235.65:42571 (local address 127.6.235.67:60745)
0928 02:11:42.274358 (+ 1206us) negotiation.cc:107] Waiting for socket to connect
0928 02:11:42.274459 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.235.65:42571: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1129,"thread_start_us":257,"threads_started":1}
W20260928 02:11:42.274941 15066 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.235.65:42571: connect: Connection refused (error 111)
I20260928 02:11:42.276558 15313 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:11:42.276219 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.235.65:42571 (local address 127.6.235.67:33801)
0928 02:11:42.276362 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0928 02:11:42.276439 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.235.65:42571: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260928 02:11:42.276868 15066 consensus_peers.cc:487] T f9b85cdcf8c04a46a8679c95bda17ac6 P baedbfdc92b946f0a07777ec1f899494 -> Peer 6e7411d0f5db463799e9b2db6669ea48 (127.6.235.65:42571): Couldn't send request to peer 6e7411d0f5db463799e9b2db6669ea48. Status: Network error: Client connection negotiation failed: client connection to 127.6.235.65:42571: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260928 02:11:42.277084 7085 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTlaUuc/build/asan/bin/kudu with pid 15050
I20260928 02:11:42.304280 7085 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskTlaUuc/build/asan/bin/kudu pid 14662
I20260928 02:11:42.305564 14734 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 02:11:42.422626 7085 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTlaUuc/build/asan/bin/kudu with pid 14662
2026-09-28T02:11:42Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipal (6917 ms)
[ RUN ] SecurityITest.TestNonExistentPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790561432251112-7085-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 02:11:42 dist-test-slave-86gv krb5kdc[15318](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 02:11:42 dist-test-slave-86gv krb5kdc[15318](info): set up 2 sockets
Sep 28 02:11:42 dist-test-slave-86gv krb5kdc[15318](info): commencing operation
krb5kdc: starting...
W20260928 02:11:43.534402 7085 mini_kdc.cc:121] Time spent starting KDC: real 1.066s user 0.000s sys 0.023s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790561432251112-7085-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790561432251112-7085-0/krb5kdc/test-user.keytab.
Sep 28 02:11:43 dist-test-slave-86gv krb5kdc[15318](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561503, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T02:11:43Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T02:11:43Z Disabled control of system clock
WARNING: no policy specified for kudu/127.6.235.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.235.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.235.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790561432251112-7085-0/krb5kdc/kudu_127.6.235.126.keytab.
Entry for principal kudu/127.6.235.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790561432251112-7085-0/krb5kdc/kudu_127.6.235.126.keytab.
I20260928 02:11:43.776954 7085 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTlaUuc/build/asan/bin/kudu
/tmp/dist-test-taskTlaUuc/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790561432251112-7085-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790561432251112-7085-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790561432251112-7085-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790561432251112-7085-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.235.126:39455
--webserver_interface=127.6.235.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790561432251112-7085-0/krb5kdc/kudu_127.6.235.126.keytab
--principal=kudu/127.6.235.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.235.84:33235
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.235.126:39455
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--principal=oryx with env {KRB5CCNAME=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790561432251112-7085-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790561432251112-7085-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790561432251112-7085-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:11:44.116082 15331 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:11:44.116385 15331 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:11:44.116475 15331 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:11:44.123780 15331 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 02:11:44.123910 15331 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:11:44.123960 15331 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:11:44.124022 15331 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 02:11:44.124063 15331 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 02:11:44.134172 15331 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.235.84:33235
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790561432251112-7085-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790561432251112-7085-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.235.126:39455
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790561432251112-7085-0/krb5kdc/kudu_127.6.235.126.keytab
--principal=oryx
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.235.126:39455
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790561432251112-7085-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.235.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790561432251112-7085-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 00:51:59 UTC on ccycloud.quasar-axgkwn.root.comops.site
build id 4676464
ASAN enabled
I20260928 02:11:44.135483 15331 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:11:44.136955 15331 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:11:44.149600 15336 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:11:44.149623 15339 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:11:44.149552 15337 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:11:44.150444 15331 server_base.cc:1034] running on GCE node
I20260928 02:11:44.151357 15331 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:11:44.152936 15331 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:11:44.154206 15331 hybrid_clock.cc:648] HybridClock initialized: now 1790561504154147 us; error 50 us; skew 500 ppm
I20260928 02:11:44.154604 15331 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Runtime error: kinit failed: unable to login from keytab: Keytab contains no suitable keys for oryx@KRBTEST.COM
2026-09-28T02:11:44Z chronyd exiting
[ OK ] SecurityITest.TestNonExistentPrincipal (1802 ms)
[ RUN ] SecurityITest.TestMismatchingPrincipals
Loading random data
Initializing database '/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790561432251112-7085-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 02:11:44 dist-test-slave-86gv krb5kdc[15345](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 02:11:44 dist-test-slave-86gv krb5kdc[15345](info): set up 2 sockets
Sep 28 02:11:44 dist-test-slave-86gv krb5kdc[15345](info): commencing operation
krb5kdc: starting...
W20260928 02:11:45.354251 7085 mini_kdc.cc:121] Time spent starting KDC: real 1.082s user 0.001s sys 0.026s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790561432251112-7085-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790561432251112-7085-0/krb5kdc/test-user.keytab.
Sep 28 02:11:45 dist-test-slave-86gv krb5kdc[15345](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561505, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T02:11:45Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T02:11:45Z Disabled control of system clock
WARNING: no policy specified for kudu/127.6.235.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.235.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.235.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790561432251112-7085-0/krb5kdc/kudu_127.6.235.126.keytab.
Entry for principal kudu/127.6.235.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790561432251112-7085-0/krb5kdc/kudu_127.6.235.126.keytab.
I20260928 02:11:45.596725 7085 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTlaUuc/build/asan/bin/kudu
/tmp/dist-test-taskTlaUuc/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790561432251112-7085-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790561432251112-7085-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790561432251112-7085-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790561432251112-7085-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.235.126:45241
--webserver_interface=127.6.235.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790561432251112-7085-0/krb5kdc/kudu_127.6.235.126.keytab
--principal=kudu/127.6.235.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.235.84:40383
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.235.126:45241
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790561432251112-7085-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790561432251112-7085-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790561432251112-7085-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:11:45.934139 15358 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:11:45.934456 15358 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:11:45.934546 15358 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:11:45.942137 15358 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 02:11:45.942258 15358 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:11:45.942303 15358 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:11:45.942359 15358 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 02:11:45.942402 15358 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 02:11:45.952232 15358 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.235.84:40383
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790561432251112-7085-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790561432251112-7085-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.235.126:45241
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790561432251112-7085-0/krb5kdc/kudu_127.6.235.126.keytab
--principal=kudu/127.6.235.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.235.126:45241
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790561432251112-7085-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.235.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790561432251112-7085-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 00:51:59 UTC on ccycloud.quasar-axgkwn.root.comops.site
build id 4676464
ASAN enabled
I20260928 02:11:45.953814 15358 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:11:45.955181 15358 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:11:45.967381 15364 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:11:45.967340 15366 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:11:45.967429 15363 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:11:45.968397 15358 server_base.cc:1034] running on GCE node
I20260928 02:11:45.969275 15358 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:11:45.970690 15358 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:11:45.972018 15358 hybrid_clock.cc:648] HybridClock initialized: now 1790561505971980 us; error 56 us; skew 500 ppm
I20260928 02:11:45.972378 15358 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 02:11:45 dist-test-slave-86gv krb5kdc[15345](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561505, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.235.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:11:45.975705 15358 init.cc:377] Logged in from keytab as kudu/127.6.235.126@KRBTEST.COM (short username kudu)
I20260928 02:11:45.977363 15358 webserver.cc:460] Webserver started at http://127.6.235.126:43177/ using document root <none> and password file <none>
I20260928 02:11:45.978058 15358 fs_manager.cc:359] Metadata directory not provided
I20260928 02:11:45.978155 15358 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:11:45.978417 15358 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:11:45.981494 15358 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790561432251112-7085-0/minicluster-data/master-0/data/instance:
uuid: "60c1c937015b40aeabd06b5888025b76"
format_stamp: "Formatted at 2026-09-28 02:11:45 on dist-test-slave-86gv"
server_key: "c40a3f8b864de51e18ccd87933ee1da0"
server_key_iv: "4ae0cc70d15e61a9f26b358c23ef7281"
server_key_version: "encryptionkey@0"
I20260928 02:11:45.982227 15358 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790561432251112-7085-0/minicluster-data/master-0/wal/instance:
uuid: "60c1c937015b40aeabd06b5888025b76"
format_stamp: "Formatted at 2026-09-28 02:11:45 on dist-test-slave-86gv"
server_key: "c40a3f8b864de51e18ccd87933ee1da0"
server_key_iv: "4ae0cc70d15e61a9f26b358c23ef7281"
server_key_version: "encryptionkey@0"
I20260928 02:11:45.987502 15358 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260928 02:11:45.991089 15373 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:11:45.992666 15358 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260928 02:11:45.992861 15358 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790561432251112-7085-0/minicluster-data/master-0/data,/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790561432251112-7085-0/minicluster-data/master-0/wal
uuid: "60c1c937015b40aeabd06b5888025b76"
format_stamp: "Formatted at 2026-09-28 02:11:45 on dist-test-slave-86gv"
server_key: "c40a3f8b864de51e18ccd87933ee1da0"
server_key_iv: "4ae0cc70d15e61a9f26b358c23ef7281"
server_key_version: "encryptionkey@0"
I20260928 02:11:45.993150 15358 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790561432251112-7085-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790561432251112-7085-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790561432251112-7085-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:11:46.009927 15358 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:11:46.014586 15358 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:11:46.014892 15358 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:11:46.033587 15358 rpc_server.cc:307] RPC server started. Bound to: 127.6.235.126:45241
I20260928 02:11:46.033610 15424 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.235.126:45241 every 8 connection(s)
I20260928 02:11:46.034989 15358 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790561432251112-7085-0/minicluster-data/master-0/data/info.pb
I20260928 02:11:46.038995 15425 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:11:46.039467 7085 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTlaUuc/build/asan/bin/kudu as pid 15358
I20260928 02:11:46.039716 7085 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790561432251112-7085-0/minicluster-data/master-0/wal/instance
I20260928 02:11:46.040514 7085 external_mini_cluster.cc:1428] Setting key ee2015a1ac67cf3432e6f25319c4378a
I20260928 02:11:46.054209 15425 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 60c1c937015b40aeabd06b5888025b76: Bootstrap starting.
Sep 28 02:11:46 dist-test-slave-86gv krb5kdc[15345](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561505, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.235.126@KRBTEST.COM
I20260928 02:11:46.058053 15425 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 60c1c937015b40aeabd06b5888025b76: Neither blocks nor log segments found. Creating new log.
I20260928 02:11:46.059290 15425 log.cc:824] T 00000000000000000000000000000000 P 60c1c937015b40aeabd06b5888025b76: Log is configured to *not* fsync() on all Append() calls
I20260928 02:11:46.062402 15425 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 60c1c937015b40aeabd06b5888025b76: No bootstrap required, opened a new log
I20260928 02:11:46.065582 15428 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:11:46.042734 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48846 (local address 127.6.235.126:45241)
0928 02:11:46.043526 (+ 792us) server_negotiation.cc:206] Beginning negotiation
0928 02:11:46.043538 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0928 02:11:46.043588 (+ 50us) server_negotiation.cc:407] Connection header received
0928 02:11:46.045937 (+ 2349us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:11:46.045967 (+ 30us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:11:46.046445 (+ 478us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:11:46.046813 (+ 368us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:11:46.048757 (+ 1944us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:11:46.050201 (+ 1444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:11:46.052088 (+ 1887us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:11:46.052994 (+ 906us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:11:46.053180 (+ 186us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:11:46.057836 (+ 4656us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:11:46.057845 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:11:46.057860 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:11:46.057906 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:11:46.061367 (+ 3461us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:11:46.062099 (+ 732us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:11:46.062106 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:11:46.062113 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:11:46.062273 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:11:46.062721 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:11:46.062726 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:11:46.062729 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:11:46.063279 (+ 550us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:11:46.063589 (+ 310us) server_negotiation.cc:1035] Waiting for connection context
0928 02:11:46.064054 (+ 465us) server_negotiation.cc:299] Negotiation successful
0928 02:11:46.064469 (+ 415us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":554,"thread_start_us":358,"threads_started":1}
I20260928 02:11:46.071465 15425 raft_consensus.cc:348] T 00000000000000000000000000000000 P 60c1c937015b40aeabd06b5888025b76 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "60c1c937015b40aeabd06b5888025b76" member_type: VOTER last_known_addr { host: "127.6.235.126" port: 45241 } }
I20260928 02:11:46.071924 15425 raft_consensus.cc:374] T 00000000000000000000000000000000 P 60c1c937015b40aeabd06b5888025b76 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:11:46.072057 15425 raft_consensus.cc:729] T 00000000000000000000000000000000 P 60c1c937015b40aeabd06b5888025b76 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 60c1c937015b40aeabd06b5888025b76, State: Initialized, Role: FOLLOWER
I20260928 02:11:46.072696 15425 consensus_queue.cc:260] T 00000000000000000000000000000000 P 60c1c937015b40aeabd06b5888025b76 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "60c1c937015b40aeabd06b5888025b76" member_type: VOTER last_known_addr { host: "127.6.235.126" port: 45241 } }
I20260928 02:11:46.072952 15425 raft_consensus.cc:388] T 00000000000000000000000000000000 P 60c1c937015b40aeabd06b5888025b76 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 02:11:46.073108 15425 raft_consensus.cc:482] T 00000000000000000000000000000000 P 60c1c937015b40aeabd06b5888025b76 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 02:11:46.073289 15425 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 60c1c937015b40aeabd06b5888025b76 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:11:46.075434 15425 raft_consensus.cc:504] T 00000000000000000000000000000000 P 60c1c937015b40aeabd06b5888025b76 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "60c1c937015b40aeabd06b5888025b76" member_type: VOTER last_known_addr { host: "127.6.235.126" port: 45241 } }
I20260928 02:11:46.075946 15425 leader_election.cc:302] T 00000000000000000000000000000000 P 60c1c937015b40aeabd06b5888025b76 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 60c1c937015b40aeabd06b5888025b76; no voters:
I20260928 02:11:46.076591 15425 leader_election.cc:288] T 00000000000000000000000000000000 P 60c1c937015b40aeabd06b5888025b76 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 02:11:46.076763 15430 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 60c1c937015b40aeabd06b5888025b76 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:11:46.077512 15430 raft_consensus.cc:686] T 00000000000000000000000000000000 P 60c1c937015b40aeabd06b5888025b76 [term 1 LEADER]: Becoming Leader. State: Replica: 60c1c937015b40aeabd06b5888025b76, State: Running, Role: LEADER
I20260928 02:11:46.078146 15430 consensus_queue.cc:237] T 00000000000000000000000000000000 P 60c1c937015b40aeabd06b5888025b76 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "60c1c937015b40aeabd06b5888025b76" member_type: VOTER last_known_addr { host: "127.6.235.126" port: 45241 } }
I20260928 02:11:46.079010 15425 sys_catalog.cc:564] T 00000000000000000000000000000000 P 60c1c937015b40aeabd06b5888025b76 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 02:11:46.081502 15432 sys_catalog.cc:455] T 00000000000000000000000000000000 P 60c1c937015b40aeabd06b5888025b76 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 60c1c937015b40aeabd06b5888025b76. Latest consensus state: current_term: 1 leader_uuid: "60c1c937015b40aeabd06b5888025b76" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "60c1c937015b40aeabd06b5888025b76" member_type: VOTER last_known_addr { host: "127.6.235.126" port: 45241 } } }
I20260928 02:11:46.081481 15431 sys_catalog.cc:455] T 00000000000000000000000000000000 P 60c1c937015b40aeabd06b5888025b76 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "60c1c937015b40aeabd06b5888025b76" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "60c1c937015b40aeabd06b5888025b76" member_type: VOTER last_known_addr { host: "127.6.235.126" port: 45241 } } }
I20260928 02:11:46.081930 15432 sys_catalog.cc:458] T 00000000000000000000000000000000 P 60c1c937015b40aeabd06b5888025b76 [sys.catalog]: This master's current role is: LEADER
I20260928 02:11:46.081962 15431 sys_catalog.cc:458] T 00000000000000000000000000000000 P 60c1c937015b40aeabd06b5888025b76 [sys.catalog]: This master's current role is: LEADER
I20260928 02:11:46.085809 15437 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 02:11:46.091598 15437 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 02:11:46.101354 15437 catalog_manager.cc:1347] Generated new cluster ID: 67eb5f65bfaf4097ac93c303c093b5b4
I20260928 02:11:46.101491 15437 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 02:11:46.112761 15437 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 02:11:46.114305 15437 catalog_manager.cc:1504] Loading token signing keys...
I20260928 02:11:46.127315 15437 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 60c1c937015b40aeabd06b5888025b76: Generated new TSK 0
I20260928 02:11:46.128475 15437 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.6.235.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.235.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.235.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790561432251112-7085-0/krb5kdc/kudu_127.6.235.65.keytab.
Entry for principal kudu/127.6.235.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790561432251112-7085-0/krb5kdc/kudu_127.6.235.65.keytab.
I20260928 02:11:46.192786 7085 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTlaUuc/build/asan/bin/kudu
/tmp/dist-test-taskTlaUuc/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790561432251112-7085-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790561432251112-7085-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790561432251112-7085-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790561432251112-7085-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.235.65:0
--local_ip_for_outbound_sockets=127.6.235.65
--webserver_interface=127.6.235.65
--webserver_port=0
--tserver_master_addrs=127.6.235.126:45241
--keytab_file=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790561432251112-7085-0/krb5kdc/kudu_127.6.235.65.keytab
--principal=kudu/127.6.235.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.235.84:40383
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790561432251112-7085-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790561432251112-7085-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790561432251112-7085-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:11:46.537137 15451 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:11:46.537420 15451 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:11:46.537624 15451 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:11:46.544984 15451 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:11:46.545091 15451 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:11:46.545261 15451 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.235.65
I20260928 02:11:46.556246 15451 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.235.84:40383
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790561432251112-7085-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790561432251112-7085-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790561432251112-7085-0/krb5kdc/kudu_127.6.235.65.keytab
--principal=kudu/127.6.235.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.235.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790561432251112-7085-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.6.235.65
--webserver_port=0
--tserver_master_addrs=127.6.235.126:45241
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.235.65
--log_dir=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790561432251112-7085-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 00:51:59 UTC on ccycloud.quasar-axgkwn.root.comops.site
build id 4676464
ASAN enabled
I20260928 02:11:46.557705 15451 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:11:46.559026 15451 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:11:46.572788 15459 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:11:46.572880 15456 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:11:46.572824 15457 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:11:46.573626 15451 server_base.cc:1034] running on GCE node
I20260928 02:11:46.574211 15451 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:11:46.575197 15451 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:11:46.576399 15451 hybrid_clock.cc:648] HybridClock initialized: now 1790561506576343 us; error 47 us; skew 500 ppm
I20260928 02:11:46.576735 15451 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 02:11:46 dist-test-slave-86gv krb5kdc[15345](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561506, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.235.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:11:46.579924 15451 init.cc:377] Logged in from keytab as kudu/127.6.235.65@KRBTEST.COM (short username kudu)
I20260928 02:11:46.581509 15451 webserver.cc:460] Webserver started at http://127.6.235.65:38879/ using document root <none> and password file <none>
I20260928 02:11:46.582190 15451 fs_manager.cc:359] Metadata directory not provided
I20260928 02:11:46.582289 15451 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:11:46.582544 15451 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:11:46.585451 15451 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790561432251112-7085-0/minicluster-data/ts-0/data/instance:
uuid: "97742803bab84b76963d080e3e694ac9"
format_stamp: "Formatted at 2026-09-28 02:11:46 on dist-test-slave-86gv"
server_key: "736bf58ba1048c1321ae92e6af543388"
server_key_iv: "85cd4b983c1cf93ab6073bc24cf12705"
server_key_version: "encryptionkey@0"
I20260928 02:11:46.586228 15451 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790561432251112-7085-0/minicluster-data/ts-0/wal/instance:
uuid: "97742803bab84b76963d080e3e694ac9"
format_stamp: "Formatted at 2026-09-28 02:11:46 on dist-test-slave-86gv"
server_key: "736bf58ba1048c1321ae92e6af543388"
server_key_iv: "85cd4b983c1cf93ab6073bc24cf12705"
server_key_version: "encryptionkey@0"
I20260928 02:11:46.591418 15451 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260928 02:11:46.594972 15466 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:11:46.596393 15451 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.001s
I20260928 02:11:46.596568 15451 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790561432251112-7085-0/minicluster-data/ts-0/data,/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790561432251112-7085-0/minicluster-data/ts-0/wal
uuid: "97742803bab84b76963d080e3e694ac9"
format_stamp: "Formatted at 2026-09-28 02:11:46 on dist-test-slave-86gv"
server_key: "736bf58ba1048c1321ae92e6af543388"
server_key_iv: "85cd4b983c1cf93ab6073bc24cf12705"
server_key_version: "encryptionkey@0"
I20260928 02:11:46.596796 15451 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790561432251112-7085-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790561432251112-7085-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790561432251112-7085-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:11:46.607815 15451 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:11:46.611797 15451 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:11:46.612110 15451 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:11:46.613230 15451 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:11:46.614614 15451 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:11:46.614717 15451 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:11:46.614872 15451 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:11:46.614945 15451 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:11:46.651307 15451 rpc_server.cc:307] RPC server started. Bound to: 127.6.235.65:36415
I20260928 02:11:46.651314 15578 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.235.65:36415 every 8 connection(s)
I20260928 02:11:46.653278 15451 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790561432251112-7085-0/minicluster-data/ts-0/data/info.pb
I20260928 02:11:46.657100 7085 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTlaUuc/build/asan/bin/kudu as pid 15451
I20260928 02:11:46.657330 7085 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790561432251112-7085-0/minicluster-data/ts-0/wal/instance
I20260928 02:11:46.657954 7085 external_mini_cluster.cc:1428] Setting key 5941dfa18b2ea6390b84b8cc857e19a2
Sep 28 02:11:46 dist-test-slave-86gv krb5kdc[15345](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561506, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.235.65@KRBTEST.COM for kudu/127.6.235.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.6.235.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.235.66@KRBTEST.COM" created.
I20260928 02:11:46.690459 15581 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:11:46.656274 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.235.65:58741 (local address 127.6.235.126:45241)
0928 02:11:46.656728 (+ 454us) server_negotiation.cc:206] Beginning negotiation
0928 02:11:46.656734 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 02:11:46.668875 (+ 12141us) server_negotiation.cc:407] Connection header received
0928 02:11:46.671976 (+ 3101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:11:46.671981 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:11:46.672087 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:11:46.672195 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:11:46.674567 (+ 2372us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:11:46.675382 (+ 815us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:11:46.678273 (+ 2891us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:11:46.678844 (+ 571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:11:46.679022 (+ 178us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:11:46.683716 (+ 4694us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:11:46.683722 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:11:46.683725 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:11:46.683760 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:11:46.686860 (+ 3100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:11:46.687763 (+ 903us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:11:46.687772 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:11:46.687779 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:11:46.687936 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:11:46.688361 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:11:46.688369 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:11:46.688375 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:11:46.688764 (+ 389us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:11:46.688938 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0928 02:11:46.689729 (+ 791us) server_negotiation.cc:299] Negotiation successful
0928 02:11:46.689981 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":325,"thread_start_us":211,"threads_started":1}
I20260928 02:11:46.691273 15582 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:11:46.656841 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.235.126:45241 (local address 127.6.235.65:58741)
0928 02:11:46.668589 (+ 11748us) negotiation.cc:107] Waiting for socket to connect
0928 02:11:46.668662 (+ 73us) client_negotiation.cc:174] Beginning negotiation
0928 02:11:46.671571 (+ 2909us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:11:46.672513 (+ 942us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:11:46.672528 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:11:46.673434 (+ 906us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:11:46.674336 (+ 902us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:11:46.674364 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:11:46.675594 (+ 1230us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:11:46.675603 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:11:46.678042 (+ 2439us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:11:46.678064 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:11:46.679129 (+ 1065us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:11:46.679138 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:11:46.679512 (+ 374us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:11:46.680465 (+ 953us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:11:46.680506 (+ 41us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:11:46.683427 (+ 2921us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:11:46.687025 (+ 3598us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:11:46.687034 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:11:46.687051 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:11:46.687604 (+ 553us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:11:46.688043 (+ 439us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:11:46.688046 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:11:46.688049 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:11:46.688236 (+ 187us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:11:46.688933 (+ 697us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:11:46.688946 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:11:46.689316 (+ 370us) client_negotiation.cc:769] Sending connection context
0928 02:11:46.689566 (+ 250us) client_negotiation.cc:240] Negotiation successful
0928 02:11:46.689977 (+ 411us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11352,"thread_start_us":12006,"threads_started":1}
I20260928 02:11:46.692960 15579 heartbeater.cc:347] Connected to a master server at 127.6.235.126:45241
I20260928 02:11:46.693318 15579 heartbeater.cc:464] Registering TS with master...
I20260928 02:11:46.694069 15579 heartbeater.cc:511] Master 127.6.235.126:45241 requested a full tablet report, sending...
I20260928 02:11:46.696506 15390 ts_manager.cc:194] Registered new tserver with Master: 97742803bab84b76963d080e3e694ac9 (127.6.235.65:36415)
I20260928 02:11:46.698212 15390 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.235.65@KRBTEST.COM'} at 127.6.235.65:58741
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.235.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790561432251112-7085-0/krb5kdc/kudu_127.6.235.66.keytab.
Entry for principal kudu/127.6.235.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790561432251112-7085-0/krb5kdc/kudu_127.6.235.66.keytab.
I20260928 02:11:46.716717 7085 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTlaUuc/build/asan/bin/kudu
/tmp/dist-test-taskTlaUuc/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790561432251112-7085-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790561432251112-7085-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790561432251112-7085-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790561432251112-7085-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.235.66:0
--local_ip_for_outbound_sockets=127.6.235.66
--webserver_interface=127.6.235.66
--webserver_port=0
--tserver_master_addrs=127.6.235.126:45241
--keytab_file=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790561432251112-7085-0/krb5kdc/kudu_127.6.235.66.keytab
--principal=kudu/127.6.235.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.235.84:40383
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790561432251112-7085-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790561432251112-7085-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790561432251112-7085-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:11:47.052421 15585 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:11:47.052794 15585 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:11:47.052999 15585 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:11:47.060357 15585 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:11:47.060526 15585 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:11:47.060719 15585 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.235.66
I20260928 02:11:47.070763 15585 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.235.84:40383
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790561432251112-7085-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790561432251112-7085-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790561432251112-7085-0/krb5kdc/kudu_127.6.235.66.keytab
--principal=kudu/127.6.235.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.235.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790561432251112-7085-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.6.235.66
--webserver_port=0
--tserver_master_addrs=127.6.235.126:45241
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.235.66
--log_dir=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790561432251112-7085-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 00:51:59 UTC on ccycloud.quasar-axgkwn.root.comops.site
build id 4676464
ASAN enabled
I20260928 02:11:47.072120 15585 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:11:47.073599 15585 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:11:47.087245 15590 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:11:47.087215 15593 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:11:47.087433 15591 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:11:47.088011 15585 server_base.cc:1034] running on GCE node
I20260928 02:11:47.088649 15585 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:11:47.089772 15585 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:11:47.091096 15585 hybrid_clock.cc:648] HybridClock initialized: now 1790561507091025 us; error 78 us; skew 500 ppm
I20260928 02:11:47.091473 15585 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 02:11:47 dist-test-slave-86gv krb5kdc[15345](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561507, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.235.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:11:47.094895 15585 init.cc:377] Logged in from keytab as kudu/127.6.235.66@KRBTEST.COM (short username kudu)
I20260928 02:11:47.096462 15585 webserver.cc:460] Webserver started at http://127.6.235.66:33929/ using document root <none> and password file <none>
I20260928 02:11:47.097250 15585 fs_manager.cc:359] Metadata directory not provided
I20260928 02:11:47.097399 15585 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:11:47.097784 15585 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:11:47.100975 15585 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790561432251112-7085-0/minicluster-data/ts-1/data/instance:
uuid: "ed97ba26f8464e418f541f35eb4d0201"
format_stamp: "Formatted at 2026-09-28 02:11:47 on dist-test-slave-86gv"
server_key: "fb74f287cb4d5ba0e2c0355d1378013a"
server_key_iv: "2096fb1f65eaadd7851e04704496906a"
server_key_version: "encryptionkey@0"
I20260928 02:11:47.101840 15585 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790561432251112-7085-0/minicluster-data/ts-1/wal/instance:
uuid: "ed97ba26f8464e418f541f35eb4d0201"
format_stamp: "Formatted at 2026-09-28 02:11:47 on dist-test-slave-86gv"
server_key: "fb74f287cb4d5ba0e2c0355d1378013a"
server_key_iv: "2096fb1f65eaadd7851e04704496906a"
server_key_version: "encryptionkey@0"
I20260928 02:11:47.107131 15585 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.004s
I20260928 02:11:47.110719 15600 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:11:47.112095 15585 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260928 02:11:47.112273 15585 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790561432251112-7085-0/minicluster-data/ts-1/data,/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790561432251112-7085-0/minicluster-data/ts-1/wal
uuid: "ed97ba26f8464e418f541f35eb4d0201"
format_stamp: "Formatted at 2026-09-28 02:11:47 on dist-test-slave-86gv"
server_key: "fb74f287cb4d5ba0e2c0355d1378013a"
server_key_iv: "2096fb1f65eaadd7851e04704496906a"
server_key_version: "encryptionkey@0"
I20260928 02:11:47.112490 15585 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790561432251112-7085-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790561432251112-7085-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790561432251112-7085-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:11:47.129962 15585 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:11:47.134044 15585 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:11:47.134351 15585 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:11:47.135309 15585 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:11:47.136672 15585 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:11:47.136777 15585 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:11:47.136973 15585 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:11:47.137064 15585 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:11:47.170789 15585 rpc_server.cc:307] RPC server started. Bound to: 127.6.235.66:36665
I20260928 02:11:47.170809 15712 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.235.66:36665 every 8 connection(s)
I20260928 02:11:47.172230 15585 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790561432251112-7085-0/minicluster-data/ts-1/data/info.pb
I20260928 02:11:47.179944 7085 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTlaUuc/build/asan/bin/kudu as pid 15585
I20260928 02:11:47.180166 7085 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790561432251112-7085-0/minicluster-data/ts-1/wal/instance
I20260928 02:11:47.180707 7085 external_mini_cluster.cc:1428] Setting key d15ed8ade167718ac8ea1f7739522b10
Sep 28 02:11:47 dist-test-slave-86gv krb5kdc[15345](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561507, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.235.66@KRBTEST.COM for kudu/127.6.235.126@KRBTEST.COM
I20260928 02:11:47.196029 15581 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:11:47.175004 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.235.66:36485 (local address 127.6.235.126:45241)
0928 02:11:47.175222 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0928 02:11:47.175229 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 02:11:47.176497 (+ 1268us) server_negotiation.cc:407] Connection header received
0928 02:11:47.178372 (+ 1875us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:11:47.178376 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:11:47.178452 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:11:47.178574 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:11:47.180527 (+ 1953us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:11:47.181475 (+ 948us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:11:47.183616 (+ 2141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:11:47.184030 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:11:47.184141 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:11:47.189149 (+ 5008us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:11:47.189155 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:11:47.189158 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:11:47.189195 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:11:47.192483 (+ 3288us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:11:47.193391 (+ 908us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:11:47.193399 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:11:47.193405 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:11:47.193574 (+ 169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:11:47.194021 (+ 447us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:11:47.194029 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:11:47.194035 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:11:47.194415 (+ 380us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:11:47.194591 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0928 02:11:47.195293 (+ 702us) server_negotiation.cc:299] Negotiation successful
0928 02:11:47.195571 (+ 278us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260928 02:11:47.197132 15715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:11:47.175534 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.235.126:45241 (local address 127.6.235.66:36485)
0928 02:11:47.176312 (+ 778us) negotiation.cc:107] Waiting for socket to connect
0928 02:11:47.176360 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0928 02:11:47.178086 (+ 1726us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:11:47.178784 (+ 698us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:11:47.178793 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:11:47.179386 (+ 593us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:11:47.180316 (+ 930us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:11:47.180348 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:11:47.181706 (+ 1358us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:11:47.181711 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:11:47.183438 (+ 1727us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:11:47.183458 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:11:47.184295 (+ 837us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:11:47.184305 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:11:47.184696 (+ 391us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:11:47.185577 (+ 881us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:11:47.185620 (+ 43us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:11:47.188809 (+ 3189us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:11:47.192682 (+ 3873us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:11:47.192690 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:11:47.192706 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:11:47.193263 (+ 557us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:11:47.193698 (+ 435us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:11:47.193702 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:11:47.193705 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:11:47.193896 (+ 191us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:11:47.194535 (+ 639us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:11:47.194541 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:11:47.194906 (+ 365us) client_negotiation.cc:769] Sending connection context
0928 02:11:47.195148 (+ 242us) client_negotiation.cc:240] Negotiation successful
0928 02:11:47.195609 (+ 461us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":525,"thread_start_us":313,"threads_started":1}
I20260928 02:11:47.198732 15713 heartbeater.cc:347] Connected to a master server at 127.6.235.126:45241
I20260928 02:11:47.199075 15713 heartbeater.cc:464] Registering TS with master...
I20260928 02:11:47.199883 15713 heartbeater.cc:511] Master 127.6.235.126:45241 requested a full tablet report, sending...
I20260928 02:11:47.201665 15390 ts_manager.cc:194] Registered new tserver with Master: ed97ba26f8464e418f541f35eb4d0201 (127.6.235.66:36665)
I20260928 02:11:47.202574 15390 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.235.66@KRBTEST.COM'} at 127.6.235.66:36485
WARNING: no policy specified for kudu/127.6.235.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.235.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.235.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790561432251112-7085-0/krb5kdc/kudu_127.6.235.67.keytab.
Entry for principal kudu/127.6.235.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790561432251112-7085-0/krb5kdc/kudu_127.6.235.67.keytab.
I20260928 02:11:47.237123 7085 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTlaUuc/build/asan/bin/kudu
/tmp/dist-test-taskTlaUuc/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790561432251112-7085-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790561432251112-7085-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790561432251112-7085-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790561432251112-7085-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.235.67:0
--local_ip_for_outbound_sockets=127.6.235.67
--webserver_interface=127.6.235.67
--webserver_port=0
--tserver_master_addrs=127.6.235.126:45241
--keytab_file=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790561432251112-7085-0/krb5kdc/kudu_127.6.235.67.keytab
--principal=kudu/127.6.235.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.235.84:40383
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790561432251112-7085-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790561432251112-7085-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790561432251112-7085-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:11:47.580802 15718 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:11:47.581223 15718 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:11:47.581400 15718 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:11:47.589213 15718 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:11:47.589365 15718 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:11:47.589550 15718 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.235.67
I20260928 02:11:47.600183 15718 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.235.84:40383
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790561432251112-7085-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790561432251112-7085-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790561432251112-7085-0/krb5kdc/kudu_127.6.235.67.keytab
--principal=kudu/127.6.235.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.235.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790561432251112-7085-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.6.235.67
--webserver_port=0
--tserver_master_addrs=127.6.235.126:45241
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.235.67
--log_dir=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790561432251112-7085-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 00:51:59 UTC on ccycloud.quasar-axgkwn.root.comops.site
build id 4676464
ASAN enabled
I20260928 02:11:47.601626 15718 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:11:47.602984 15718 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:11:47.617358 15726 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:11:47.617309 15724 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:11:47.617336 15723 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:11:47.617997 15718 server_base.cc:1034] running on GCE node
I20260928 02:11:47.618709 15718 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:11:47.619871 15718 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:11:47.621101 15718 hybrid_clock.cc:648] HybridClock initialized: now 1790561507621048 us; error 49 us; skew 500 ppm
I20260928 02:11:47.621438 15718 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 02:11:47 dist-test-slave-86gv krb5kdc[15345](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561507, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.235.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:11:47.624606 15718 init.cc:377] Logged in from keytab as kudu/127.6.235.67@KRBTEST.COM (short username kudu)
I20260928 02:11:47.626214 15718 webserver.cc:460] Webserver started at http://127.6.235.67:39059/ using document root <none> and password file <none>
I20260928 02:11:47.626955 15718 fs_manager.cc:359] Metadata directory not provided
I20260928 02:11:47.627102 15718 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:11:47.627424 15718 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:11:47.630570 15718 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790561432251112-7085-0/minicluster-data/ts-2/data/instance:
uuid: "f43abb731c2347e8ac70b9145993b103"
format_stamp: "Formatted at 2026-09-28 02:11:47 on dist-test-slave-86gv"
server_key: "5f42e5d7416c82c608dc686e1e6df19e"
server_key_iv: "25b74e3cdd46451a38cd083930f3d937"
server_key_version: "encryptionkey@0"
I20260928 02:11:47.631369 15718 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790561432251112-7085-0/minicluster-data/ts-2/wal/instance:
uuid: "f43abb731c2347e8ac70b9145993b103"
format_stamp: "Formatted at 2026-09-28 02:11:47 on dist-test-slave-86gv"
server_key: "5f42e5d7416c82c608dc686e1e6df19e"
server_key_iv: "25b74e3cdd46451a38cd083930f3d937"
server_key_version: "encryptionkey@0"
I20260928 02:11:47.636813 15718 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.003s
I20260928 02:11:47.640556 15733 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:11:47.642076 15718 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 02:11:47.642269 15718 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790561432251112-7085-0/minicluster-data/ts-2/data,/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790561432251112-7085-0/minicluster-data/ts-2/wal
uuid: "f43abb731c2347e8ac70b9145993b103"
format_stamp: "Formatted at 2026-09-28 02:11:47 on dist-test-slave-86gv"
server_key: "5f42e5d7416c82c608dc686e1e6df19e"
server_key_iv: "25b74e3cdd46451a38cd083930f3d937"
server_key_version: "encryptionkey@0"
I20260928 02:11:47.642480 15718 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790561432251112-7085-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790561432251112-7085-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790561432251112-7085-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:11:47.670490 15718 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:11:47.674961 15718 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:11:47.675262 15718 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:11:47.676280 15718 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:11:47.677737 15718 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:11:47.677841 15718 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:11:47.677995 15718 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:11:47.678067 15718 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:11:47.701592 15579 heartbeater.cc:503] Master 127.6.235.126:45241 was elected leader, sending a full tablet report...
I20260928 02:11:47.714035 15718 rpc_server.cc:307] RPC server started. Bound to: 127.6.235.67:38441
I20260928 02:11:47.714139 15845 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.235.67:38441 every 8 connection(s)
I20260928 02:11:47.715399 15718 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790561432251112-7085-0/minicluster-data/ts-2/data/info.pb
I20260928 02:11:47.721503 7085 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTlaUuc/build/asan/bin/kudu as pid 15718
I20260928 02:11:47.721741 7085 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790561432251112-7085-0/minicluster-data/ts-2/wal/instance
I20260928 02:11:47.722322 7085 external_mini_cluster.cc:1428] Setting key 7568cffd6b46a8ec22f642443447dbb4
Sep 28 02:11:47 dist-test-slave-86gv krb5kdc[15345](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561507, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.235.67@KRBTEST.COM for kudu/127.6.235.126@KRBTEST.COM
I20260928 02:11:47.737931 15848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:11:47.718453 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.235.67:33779 (local address 127.6.235.126:45241)
0928 02:11:47.718918 (+ 465us) server_negotiation.cc:206] Beginning negotiation
0928 02:11:47.718923 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 02:11:47.719950 (+ 1027us) server_negotiation.cc:407] Connection header received
0928 02:11:47.722324 (+ 2374us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:11:47.722328 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:11:47.722469 (+ 141us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:11:47.722567 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:11:47.724704 (+ 2137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:11:47.725857 (+ 1153us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:11:47.727545 (+ 1688us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:11:47.728037 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:11:47.728163 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:11:47.731712 (+ 3549us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:11:47.731717 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:11:47.731720 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:11:47.731752 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:11:47.734356 (+ 2604us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:11:47.735121 (+ 765us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:11:47.735129 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:11:47.735133 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:11:47.735261 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:11:47.735831 (+ 570us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:11:47.735836 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:11:47.735838 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:11:47.736255 (+ 417us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:11:47.736414 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0928 02:11:47.737312 (+ 898us) server_negotiation.cc:299] Negotiation successful
0928 02:11:47.737547 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":358,"thread_start_us":240,"threads_started":1}
I20260928 02:11:47.738992 15849 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:11:47.719093 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.235.126:45241 (local address 127.6.235.67:33779)
0928 02:11:47.719782 (+ 689us) negotiation.cc:107] Waiting for socket to connect
0928 02:11:47.719821 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0928 02:11:47.721951 (+ 2130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:11:47.722889 (+ 938us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:11:47.722902 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:11:47.723616 (+ 714us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:11:47.724522 (+ 906us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:11:47.724548 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:11:47.726005 (+ 1457us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:11:47.726009 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:11:47.727376 (+ 1367us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:11:47.727394 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:11:47.728258 (+ 864us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:11:47.728263 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:11:47.728473 (+ 210us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:11:47.729266 (+ 793us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:11:47.729289 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:11:47.731502 (+ 2213us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:11:47.734496 (+ 2994us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:11:47.734504 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:11:47.734518 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:11:47.734958 (+ 440us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:11:47.735404 (+ 446us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:11:47.735411 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:11:47.735417 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:11:47.735610 (+ 193us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:11:47.736388 (+ 778us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:11:47.736396 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:11:47.736859 (+ 463us) client_negotiation.cc:769] Sending connection context
0928 02:11:47.737215 (+ 356us) client_negotiation.cc:240] Negotiation successful
0928 02:11:47.737782 (+ 567us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":455,"thread_start_us":263,"threads_started":1}
I20260928 02:11:47.740351 15846 heartbeater.cc:347] Connected to a master server at 127.6.235.126:45241
I20260928 02:11:47.740697 15846 heartbeater.cc:464] Registering TS with master...
I20260928 02:11:47.741463 15846 heartbeater.cc:511] Master 127.6.235.126:45241 requested a full tablet report, sending...
I20260928 02:11:47.742844 15390 ts_manager.cc:194] Registered new tserver with Master: f43abb731c2347e8ac70b9145993b103 (127.6.235.67:38441)
I20260928 02:11:47.743661 15390 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.235.67@KRBTEST.COM'} at 127.6.235.67:33779
I20260928 02:11:47.750371 7085 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.6.235.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.6.235.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.6.235.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790561432251112-7085-0/krb5kdc/oryx_127.6.235.65.keytab.
Entry for principal oryx/127.6.235.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790561432251112-7085-0/krb5kdc/oryx_127.6.235.65.keytab.
WARNING: no policy specified for oryx/127.6.235.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.6.235.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.6.235.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790561432251112-7085-0/krb5kdc/oryx_127.6.235.66.keytab.
Entry for principal oryx/127.6.235.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790561432251112-7085-0/krb5kdc/oryx_127.6.235.66.keytab.
WARNING: no policy specified for oryx/127.6.235.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.6.235.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.6.235.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790561432251112-7085-0/krb5kdc/oryx_127.6.235.67.keytab.
Entry for principal oryx/127.6.235.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790561432251112-7085-0/krb5kdc/oryx_127.6.235.67.keytab.
I20260928 02:11:47.901499 7085 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskTlaUuc/build/asan/bin/kudu pid 15451
Sep 28 02:11:47 dist-test-slave-86gv krb5kdc[15345](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561505, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.235.65@KRBTEST.COM
I20260928 02:11:47.918346 15857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:11:47.902796 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42310 (local address 127.6.235.65:36415)
0928 02:11:47.903508 (+ 712us) server_negotiation.cc:206] Beginning negotiation
0928 02:11:47.903515 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 02:11:47.903543 (+ 28us) server_negotiation.cc:407] Connection header received
0928 02:11:47.903604 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:11:47.903610 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:11:47.903824 (+ 214us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:11:47.903960 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:11:47.905383 (+ 1423us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:11:47.906472 (+ 1089us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:11:47.907722 (+ 1250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:11:47.908308 (+ 586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:11:47.908415 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:11:47.912497 (+ 4082us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:11:47.912509 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:11:47.912524 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:11:47.912564 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:11:47.915507 (+ 2943us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:11:47.916282 (+ 775us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:11:47.916289 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:11:47.916291 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:11:47.916452 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:11:47.916818 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:11:47.916828 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:11:47.916831 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:11:47.917241 (+ 410us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:11:47.917460 (+ 219us) server_negotiation.cc:1035] Waiting for connection context
0928 02:11:47.917734 (+ 274us) server_negotiation.cc:299] Negotiation successful
0928 02:11:47.917926 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":529,"thread_start_us":381,"threads_started":1}
I20260928 02:11:47.919624 15574 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 02:11:48.005108 7085 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTlaUuc/build/asan/bin/kudu with pid 15451
I20260928 02:11:48.023202 7085 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskTlaUuc/build/asan/bin/kudu pid 15585
Sep 28 02:11:48 dist-test-slave-86gv krb5kdc[15345](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561505, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.235.66@KRBTEST.COM
I20260928 02:11:48.040481 15859 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:11:48.024120 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56420 (local address 127.6.235.66:36665)
0928 02:11:48.024696 (+ 576us) server_negotiation.cc:206] Beginning negotiation
0928 02:11:48.024704 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 02:11:48.024731 (+ 27us) server_negotiation.cc:407] Connection header received
0928 02:11:48.024801 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:11:48.024807 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:11:48.025085 (+ 278us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:11:48.025222 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:11:48.026483 (+ 1261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:11:48.027525 (+ 1042us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:11:48.028786 (+ 1261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:11:48.029426 (+ 640us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:11:48.029536 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:11:48.033846 (+ 4310us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:11:48.033863 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:11:48.033883 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:11:48.033927 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:11:48.037544 (+ 3617us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:11:48.038260 (+ 716us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:11:48.038270 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:11:48.038274 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:11:48.038420 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:11:48.038777 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:11:48.038783 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:11:48.038785 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:11:48.039255 (+ 470us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:11:48.039536 (+ 281us) server_negotiation.cc:1035] Waiting for connection context
0928 02:11:48.039806 (+ 270us) server_negotiation.cc:299] Negotiation successful
0928 02:11:48.040076 (+ 270us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":429,"thread_start_us":328,"threads_started":1}
I20260928 02:11:48.042232 15708 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 02:11:48.125244 7085 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTlaUuc/build/asan/bin/kudu with pid 15585
I20260928 02:11:48.141737 7085 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskTlaUuc/build/asan/bin/kudu pid 15718
Sep 28 02:11:48 dist-test-slave-86gv krb5kdc[15345](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561505, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.235.67@KRBTEST.COM
I20260928 02:11:48.158751 15861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:11:48.142705 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53184 (local address 127.6.235.67:38441)
0928 02:11:48.143239 (+ 534us) server_negotiation.cc:206] Beginning negotiation
0928 02:11:48.143247 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 02:11:48.143277 (+ 30us) server_negotiation.cc:407] Connection header received
0928 02:11:48.143338 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:11:48.143344 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:11:48.143563 (+ 219us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:11:48.143695 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:11:48.145143 (+ 1448us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:11:48.146344 (+ 1201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:11:48.147764 (+ 1420us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:11:48.148331 (+ 567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:11:48.148429 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:11:48.152993 (+ 4564us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:11:48.153007 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:11:48.153023 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:11:48.153091 (+ 68us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:11:48.155870 (+ 2779us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:11:48.156554 (+ 684us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:11:48.156561 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:11:48.156563 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:11:48.156690 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:11:48.157294 (+ 604us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:11:48.157297 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:11:48.157299 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:11:48.157650 (+ 351us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:11:48.157872 (+ 222us) server_negotiation.cc:1035] Waiting for connection context
0928 02:11:48.158226 (+ 354us) server_negotiation.cc:299] Negotiation successful
0928 02:11:48.158445 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":383,"thread_start_us":244,"threads_started":1}
I20260928 02:11:48.160182 15841 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 02:11:48.243605 7085 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTlaUuc/build/asan/bin/kudu with pid 15718
I20260928 02:11:48.260573 7085 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskTlaUuc/build/asan/bin/kudu pid 15358
I20260928 02:11:48.261825 15420 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 02:11:48.366611 7085 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTlaUuc/build/asan/bin/kudu with pid 15358
I20260928 02:11:48.388398 7085 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTlaUuc/build/asan/bin/kudu
/tmp/dist-test-taskTlaUuc/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790561432251112-7085-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790561432251112-7085-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790561432251112-7085-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790561432251112-7085-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.235.126:45241
--webserver_interface=127.6.235.126
--webserver_port=43177
--keytab_file=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790561432251112-7085-0/krb5kdc/kudu_127.6.235.126.keytab
--principal=kudu/127.6.235.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.235.84:40383
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.235.126:45241
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790561432251112-7085-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790561432251112-7085-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790561432251112-7085-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:11:48.732738 15864 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:11:48.733085 15864 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:11:48.733278 15864 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:11:48.740707 15864 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 02:11:48.740826 15864 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:11:48.740988 15864 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:11:48.741122 15864 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 02:11:48.741204 15864 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 02:11:48.751515 15864 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.235.84:40383
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790561432251112-7085-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790561432251112-7085-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.235.126:45241
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790561432251112-7085-0/krb5kdc/kudu_127.6.235.126.keytab
--principal=kudu/127.6.235.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.235.126:45241
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790561432251112-7085-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.235.126
--webserver_port=43177
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790561432251112-7085-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 00:51:59 UTC on ccycloud.quasar-axgkwn.root.comops.site
build id 4676464
ASAN enabled
I20260928 02:11:48.753000 15864 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:11:48.754367 15864 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:11:48.767098 15869 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:11:48.767086 15870 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:11:48.767092 15872 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:11:48.767777 15864 server_base.cc:1034] running on GCE node
I20260928 02:11:48.768731 15864 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:11:48.770375 15864 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:11:48.771639 15864 hybrid_clock.cc:648] HybridClock initialized: now 1790561508771571 us; error 73 us; skew 500 ppm
I20260928 02:11:48.772020 15864 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 02:11:48 dist-test-slave-86gv krb5kdc[15345](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561508, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.235.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:11:48.775296 15864 init.cc:377] Logged in from keytab as kudu/127.6.235.126@KRBTEST.COM (short username kudu)
I20260928 02:11:48.776839 15864 webserver.cc:460] Webserver started at http://127.6.235.126:43177/ using document root <none> and password file <none>
I20260928 02:11:48.777590 15864 fs_manager.cc:359] Metadata directory not provided
I20260928 02:11:48.777704 15864 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:11:48.783305 15864 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.002s
I20260928 02:11:48.786559 15879 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:11:48.788426 15864 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260928 02:11:48.788681 15864 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790561432251112-7085-0/minicluster-data/master-0/data,/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790561432251112-7085-0/minicluster-data/master-0/wal
uuid: "60c1c937015b40aeabd06b5888025b76"
format_stamp: "Formatted at 2026-09-28 02:11:45 on dist-test-slave-86gv"
server_key: "c40a3f8b864de51e18ccd87933ee1da0"
server_key_iv: "4ae0cc70d15e61a9f26b358c23ef7281"
server_key_version: "encryptionkey@0"
I20260928 02:11:48.789744 15864 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790561432251112-7085-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790561432251112-7085-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790561432251112-7085-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:11:48.818791 15864 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:11:48.823344 15864 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:11:48.823639 15864 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:11:48.843186 15864 rpc_server.cc:307] RPC server started. Bound to: 127.6.235.126:45241
I20260928 02:11:48.843205 15930 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.235.126:45241 every 8 connection(s)
I20260928 02:11:48.844782 15864 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790561432251112-7085-0/minicluster-data/master-0/data/info.pb
I20260928 02:11:48.853017 15931 sys_catalog.cc:263] Verifying existing consensus state
I20260928 02:11:48.853142 7085 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTlaUuc/build/asan/bin/kudu as pid 15864
I20260928 02:11:48.854168 7085 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTlaUuc/build/asan/bin/kudu
/tmp/dist-test-taskTlaUuc/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790561432251112-7085-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790561432251112-7085-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790561432251112-7085-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790561432251112-7085-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.235.65:36415
--local_ip_for_outbound_sockets=127.6.235.65
--tserver_master_addrs=127.6.235.126:45241
--webserver_port=38879
--webserver_interface=127.6.235.65
--keytab_file=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790561432251112-7085-0/krb5kdc/kudu_127.6.235.65.keytab
--principal=kudu/127.6.235.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.235.84:40383
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.6.235.65
--keytab_file=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790561432251112-7085-0/krb5kdc/oryx_127.6.235.65.keytab with env {KRB5CCNAME=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790561432251112-7085-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790561432251112-7085-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790561432251112-7085-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 02:11:48.858898 15931 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 60c1c937015b40aeabd06b5888025b76: Bootstrap starting.
I20260928 02:11:48.881680 15931 log.cc:824] T 00000000000000000000000000000000 P 60c1c937015b40aeabd06b5888025b76: Log is configured to *not* fsync() on all Append() calls
I20260928 02:11:48.891692 15931 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 60c1c937015b40aeabd06b5888025b76: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 02:11:48.892274 15931 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 60c1c937015b40aeabd06b5888025b76: Bootstrap complete.
I20260928 02:11:48.901803 15931 raft_consensus.cc:348] T 00000000000000000000000000000000 P 60c1c937015b40aeabd06b5888025b76 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "60c1c937015b40aeabd06b5888025b76" member_type: VOTER last_known_addr { host: "127.6.235.126" port: 45241 } }
I20260928 02:11:48.902657 15931 raft_consensus.cc:729] T 00000000000000000000000000000000 P 60c1c937015b40aeabd06b5888025b76 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 60c1c937015b40aeabd06b5888025b76, State: Initialized, Role: FOLLOWER
I20260928 02:11:48.903328 15931 consensus_queue.cc:260] T 00000000000000000000000000000000 P 60c1c937015b40aeabd06b5888025b76 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "60c1c937015b40aeabd06b5888025b76" member_type: VOTER last_known_addr { host: "127.6.235.126" port: 45241 } }
I20260928 02:11:48.903568 15931 raft_consensus.cc:388] T 00000000000000000000000000000000 P 60c1c937015b40aeabd06b5888025b76 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 02:11:48.903680 15931 raft_consensus.cc:482] T 00000000000000000000000000000000 P 60c1c937015b40aeabd06b5888025b76 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 02:11:48.903870 15931 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 60c1c937015b40aeabd06b5888025b76 [term 1 FOLLOWER]: Advancing to term 2
I20260928 02:11:48.907703 15931 raft_consensus.cc:504] T 00000000000000000000000000000000 P 60c1c937015b40aeabd06b5888025b76 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "60c1c937015b40aeabd06b5888025b76" member_type: VOTER last_known_addr { host: "127.6.235.126" port: 45241 } }
I20260928 02:11:48.908223 15931 leader_election.cc:302] T 00000000000000000000000000000000 P 60c1c937015b40aeabd06b5888025b76 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 60c1c937015b40aeabd06b5888025b76; no voters:
I20260928 02:11:48.908870 15931 leader_election.cc:288] T 00000000000000000000000000000000 P 60c1c937015b40aeabd06b5888025b76 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260928 02:11:48.909014 15935 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 60c1c937015b40aeabd06b5888025b76 [term 2 FOLLOWER]: Leader election won for term 2
I20260928 02:11:48.910400 15935 raft_consensus.cc:686] T 00000000000000000000000000000000 P 60c1c937015b40aeabd06b5888025b76 [term 2 LEADER]: Becoming Leader. State: Replica: 60c1c937015b40aeabd06b5888025b76, State: Running, Role: LEADER
I20260928 02:11:48.911054 15935 consensus_queue.cc:237] T 00000000000000000000000000000000 P 60c1c937015b40aeabd06b5888025b76 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "60c1c937015b40aeabd06b5888025b76" member_type: VOTER last_known_addr { host: "127.6.235.126" port: 45241 } }
I20260928 02:11:48.911311 15931 sys_catalog.cc:564] T 00000000000000000000000000000000 P 60c1c937015b40aeabd06b5888025b76 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 02:11:48.914037 15937 sys_catalog.cc:455] T 00000000000000000000000000000000 P 60c1c937015b40aeabd06b5888025b76 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 60c1c937015b40aeabd06b5888025b76. Latest consensus state: current_term: 2 leader_uuid: "60c1c937015b40aeabd06b5888025b76" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "60c1c937015b40aeabd06b5888025b76" member_type: VOTER last_known_addr { host: "127.6.235.126" port: 45241 } } }
I20260928 02:11:48.914124 15936 sys_catalog.cc:455] T 00000000000000000000000000000000 P 60c1c937015b40aeabd06b5888025b76 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "60c1c937015b40aeabd06b5888025b76" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "60c1c937015b40aeabd06b5888025b76" member_type: VOTER last_known_addr { host: "127.6.235.126" port: 45241 } } }
I20260928 02:11:48.914427 15937 sys_catalog.cc:458] T 00000000000000000000000000000000 P 60c1c937015b40aeabd06b5888025b76 [sys.catalog]: This master's current role is: LEADER
I20260928 02:11:48.914507 15936 sys_catalog.cc:458] T 00000000000000000000000000000000 P 60c1c937015b40aeabd06b5888025b76 [sys.catalog]: This master's current role is: LEADER
I20260928 02:11:48.922457 15940 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 02:11:48.930857 15940 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 02:11:48.933207 15940 catalog_manager.cc:1259] Loaded cluster ID: 67eb5f65bfaf4097ac93c303c093b5b4
I20260928 02:11:48.933472 15940 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 02:11:48.939662 15940 catalog_manager.cc:1504] Loading token signing keys...
I20260928 02:11:48.942299 15940 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 60c1c937015b40aeabd06b5888025b76: Loaded TSK: 0
I20260928 02:11:48.944703 15940 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260928 02:11:49.231555 15933 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:11:49.231935 15933 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:11:49.232095 15933 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:11:49.239693 15933 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:11:49.239869 15933 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:11:49.240051 15933 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.235.65
I20260928 02:11:49.250917 15933 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.235.84:40383
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790561432251112-7085-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790561432251112-7085-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790561432251112-7085-0/krb5kdc/oryx_127.6.235.65.keytab
--principal=oryx/127.6.235.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.235.65:36415
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790561432251112-7085-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.6.235.65
--webserver_port=38879
--tserver_master_addrs=127.6.235.126:45241
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.235.65
--log_dir=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790561432251112-7085-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 00:51:59 UTC on ccycloud.quasar-axgkwn.root.comops.site
build id 4676464
ASAN enabled
I20260928 02:11:49.252348 15933 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:11:49.253818 15933 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:11:49.268725 15957 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:11:49.268913 15960 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:11:49.269078 15958 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:11:49.269547 15933 server_base.cc:1034] running on GCE node
I20260928 02:11:49.270406 15933 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:11:49.271479 15933 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:11:49.272766 15933 hybrid_clock.cc:648] HybridClock initialized: now 1790561509272700 us; error 57 us; skew 500 ppm
I20260928 02:11:49.273272 15933 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 02:11:49 dist-test-slave-86gv krb5kdc[15345](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561509, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.235.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:11:49.284483 15933 init.cc:377] Logged in from keytab as oryx/127.6.235.65@KRBTEST.COM (short username oryx)
I20260928 02:11:49.286643 15933 webserver.cc:460] Webserver started at http://127.6.235.65:38879/ using document root <none> and password file <none>
I20260928 02:11:49.287452 15933 fs_manager.cc:359] Metadata directory not provided
I20260928 02:11:49.287624 15933 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:11:49.293988 15933 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.000s sys 0.003s
I20260928 02:11:49.297327 15967 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:11:49.299031 15933 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260928 02:11:49.299223 15933 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790561432251112-7085-0/minicluster-data/ts-0/data,/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790561432251112-7085-0/minicluster-data/ts-0/wal
uuid: "97742803bab84b76963d080e3e694ac9"
format_stamp: "Formatted at 2026-09-28 02:11:46 on dist-test-slave-86gv"
server_key: "736bf58ba1048c1321ae92e6af543388"
server_key_iv: "85cd4b983c1cf93ab6073bc24cf12705"
server_key_version: "encryptionkey@0"
I20260928 02:11:49.300345 15933 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790561432251112-7085-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790561432251112-7085-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790561432251112-7085-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:11:49.372709 15933 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:11:49.377836 15933 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:11:49.378155 15933 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:11:49.379252 15933 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:11:49.380759 15933 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:11:49.380856 15933 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:11:49.381043 15933 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:11:49.381161 15933 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:11:49.418068 15933 rpc_server.cc:307] RPC server started. Bound to: 127.6.235.65:36415
I20260928 02:11:49.418166 16079 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.235.65:36415 every 8 connection(s)
I20260928 02:11:49.419723 15933 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790561432251112-7085-0/minicluster-data/ts-0/data/info.pb
I20260928 02:11:49.423399 7085 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTlaUuc/build/asan/bin/kudu as pid 15933
I20260928 02:11:49.424413 7085 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTlaUuc/build/asan/bin/kudu
/tmp/dist-test-taskTlaUuc/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790561432251112-7085-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790561432251112-7085-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790561432251112-7085-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790561432251112-7085-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.235.66:36665
--local_ip_for_outbound_sockets=127.6.235.66
--tserver_master_addrs=127.6.235.126:45241
--webserver_port=33929
--webserver_interface=127.6.235.66
--keytab_file=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790561432251112-7085-0/krb5kdc/kudu_127.6.235.66.keytab
--principal=kudu/127.6.235.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.235.84:40383
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.6.235.66
--keytab_file=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790561432251112-7085-0/krb5kdc/oryx_127.6.235.66.keytab with env {KRB5CCNAME=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790561432251112-7085-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790561432251112-7085-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790561432251112-7085-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 28 02:11:49 dist-test-slave-86gv krb5kdc[15345](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.235.65@KRBTEST.COM for oryx/127.6.235.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 02:11:49 dist-test-slave-86gv krb5kdc[15345](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.235.65@KRBTEST.COM for oryx/127.6.235.126@KRBTEST.COM, Server not found in Kerberos database
W20260928 02:11:49.796453 16084 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:11:49.796801 16084 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:11:49.797029 16084 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:11:49.805831 16084 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:11:49.806010 16084 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:11:49.806203 16084 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.235.66
I20260928 02:11:49.817327 16084 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.235.84:40383
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790561432251112-7085-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790561432251112-7085-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790561432251112-7085-0/krb5kdc/oryx_127.6.235.66.keytab
--principal=oryx/127.6.235.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.235.66:36665
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790561432251112-7085-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.6.235.66
--webserver_port=33929
--tserver_master_addrs=127.6.235.126:45241
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.235.66
--log_dir=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790561432251112-7085-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 00:51:59 UTC on ccycloud.quasar-axgkwn.root.comops.site
build id 4676464
ASAN enabled
I20260928 02:11:49.818794 16084 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:11:49.820153 16084 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:11:49.838488 16090 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:11:49.838907 16091 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:11:49.840322 16093 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:11:49.841233 16084 server_base.cc:1034] running on GCE node
I20260928 02:11:49.841969 16084 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:11:49.843184 16084 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:11:49.844461 16084 hybrid_clock.cc:648] HybridClock initialized: now 1790561509844380 us; error 68 us; skew 500 ppm
I20260928 02:11:49.844826 16084 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 02:11:49 dist-test-slave-86gv krb5kdc[15345](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561509, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.235.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:11:49.848174 16084 init.cc:377] Logged in from keytab as oryx/127.6.235.66@KRBTEST.COM (short username oryx)
I20260928 02:11:49.854346 16084 webserver.cc:460] Webserver started at http://127.6.235.66:33929/ using document root <none> and password file <none>
I20260928 02:11:49.855158 16084 fs_manager.cc:359] Metadata directory not provided
I20260928 02:11:49.855326 16084 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:11:49.861191 16084 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.001s
I20260928 02:11:49.864127 16100 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:11:49.865638 16084 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 02:11:49.865818 16084 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790561432251112-7085-0/minicluster-data/ts-1/data,/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790561432251112-7085-0/minicluster-data/ts-1/wal
uuid: "ed97ba26f8464e418f541f35eb4d0201"
format_stamp: "Formatted at 2026-09-28 02:11:47 on dist-test-slave-86gv"
server_key: "fb74f287cb4d5ba0e2c0355d1378013a"
server_key_iv: "2096fb1f65eaadd7851e04704496906a"
server_key_version: "encryptionkey@0"
I20260928 02:11:49.866817 16084 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790561432251112-7085-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790561432251112-7085-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790561432251112-7085-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:11:49.885911 16084 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:11:49.890692 16084 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:11:49.890995 16084 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:11:49.892120 16084 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:11:49.893713 16084 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:11:49.893833 16084 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:11:49.893991 16084 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:11:49.894084 16084 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:11:49.938095 16084 rpc_server.cc:307] RPC server started. Bound to: 127.6.235.66:36665
I20260928 02:11:49.938135 16212 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.235.66:36665 every 8 connection(s)
I20260928 02:11:49.939640 16084 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790561432251112-7085-0/minicluster-data/ts-1/data/info.pb
I20260928 02:11:49.940448 7085 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTlaUuc/build/asan/bin/kudu as pid 16084
I20260928 02:11:49.941507 7085 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTlaUuc/build/asan/bin/kudu
/tmp/dist-test-taskTlaUuc/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790561432251112-7085-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790561432251112-7085-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790561432251112-7085-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790561432251112-7085-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.235.67:38441
--local_ip_for_outbound_sockets=127.6.235.67
--tserver_master_addrs=127.6.235.126:45241
--webserver_port=39059
--webserver_interface=127.6.235.67
--keytab_file=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790561432251112-7085-0/krb5kdc/kudu_127.6.235.67.keytab
--principal=kudu/127.6.235.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.235.84:40383
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.6.235.67
--keytab_file=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790561432251112-7085-0/krb5kdc/oryx_127.6.235.67.keytab with env {KRB5CCNAME=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790561432251112-7085-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790561432251112-7085-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790561432251112-7085-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 28 02:11:49 dist-test-slave-86gv krb5kdc[15345](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.235.66@KRBTEST.COM for oryx/127.6.235.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 02:11:49 dist-test-slave-86gv krb5kdc[15345](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.235.66@KRBTEST.COM for oryx/127.6.235.126@KRBTEST.COM, Server not found in Kerberos database
W20260928 02:11:50.311126 16217 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:11:50.311471 16217 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:11:50.311702 16217 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:11:50.319751 16217 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:11:50.319900 16217 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:11:50.320096 16217 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.235.67
I20260928 02:11:50.330546 16217 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.235.84:40383
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790561432251112-7085-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790561432251112-7085-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790561432251112-7085-0/krb5kdc/oryx_127.6.235.67.keytab
--principal=oryx/127.6.235.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.235.67:38441
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790561432251112-7085-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.6.235.67
--webserver_port=39059
--tserver_master_addrs=127.6.235.126:45241
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.235.67
--log_dir=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790561432251112-7085-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 00:51:59 UTC on ccycloud.quasar-axgkwn.root.comops.site
build id 4676464
ASAN enabled
I20260928 02:11:50.331938 16217 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:11:50.333336 16217 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:11:50.350353 16223 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:11:50.350749 16224 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:11:50.362836 16226 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:11:50.363385 16217 server_base.cc:1034] running on GCE node
I20260928 02:11:50.364943 16217 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:11:50.370839 16217 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:11:50.374471 16217 hybrid_clock.cc:648] HybridClock initialized: now 1790561510374363 us; error 93 us; skew 500 ppm
I20260928 02:11:50.375141 16217 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 02:11:50 dist-test-slave-86gv krb5kdc[15345](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561510, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.235.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:11:50.381846 16217 init.cc:377] Logged in from keytab as oryx/127.6.235.67@KRBTEST.COM (short username oryx)
I20260928 02:11:50.384842 16217 webserver.cc:460] Webserver started at http://127.6.235.67:39059/ using document root <none> and password file <none>
I20260928 02:11:50.386309 16217 fs_manager.cc:359] Metadata directory not provided
I20260928 02:11:50.386536 16217 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:11:50.397691 16217 fs_manager.cc:709] Time spent opening directory manager: real 0.007s user 0.006s sys 0.001s
I20260928 02:11:50.403789 16233 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:11:50.407058 16217 fs_manager.cc:725] Time spent opening block manager: real 0.006s user 0.006s sys 0.000s
I20260928 02:11:50.407383 16217 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790561432251112-7085-0/minicluster-data/ts-2/data,/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790561432251112-7085-0/minicluster-data/ts-2/wal
uuid: "f43abb731c2347e8ac70b9145993b103"
format_stamp: "Formatted at 2026-09-28 02:11:47 on dist-test-slave-86gv"
server_key: "5f42e5d7416c82c608dc686e1e6df19e"
server_key_iv: "25b74e3cdd46451a38cd083930f3d937"
server_key_version: "encryptionkey@0"
I20260928 02:11:50.409544 16217 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790561432251112-7085-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790561432251112-7085-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790561432251112-7085-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:11:50.465199 16217 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:11:50.474048 16217 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:11:50.474571 16217 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:11:50.476423 16217 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:11:50.479104 16217 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:11:50.479326 16217 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.001s sys 0.000s
I20260928 02:11:50.479547 16217 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:11:50.479732 16217 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:11:50.547091 16217 rpc_server.cc:307] RPC server started. Bound to: 127.6.235.67:38441
I20260928 02:11:50.547215 16345 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.235.67:38441 every 8 connection(s)
I20260928 02:11:50.550005 16217 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790561432251112-7085-0/minicluster-data/ts-2/data/info.pb
I20260928 02:11:50.556367 7085 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTlaUuc/build/asan/bin/kudu as pid 16217
Sep 28 02:11:50 dist-test-slave-86gv krb5kdc[15345](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.235.67@KRBTEST.COM for oryx/127.6.235.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 02:11:50 dist-test-slave-86gv krb5kdc[15345](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.235.67@KRBTEST.COM for oryx/127.6.235.126@KRBTEST.COM, Server not found in Kerberos database
I20260928 02:11:50.608943 16351 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:11:50.566201 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48860 (local address 127.6.235.126:45241)
0928 02:11:50.577465 (+ 11264us) server_negotiation.cc:206] Beginning negotiation
0928 02:11:50.577475 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 02:11:50.577504 (+ 29us) server_negotiation.cc:407] Connection header received
0928 02:11:50.577582 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:11:50.577590 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:11:50.577696 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:11:50.577847 (+ 151us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:11:50.581232 (+ 3385us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:11:50.584590 (+ 3358us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:11:50.587507 (+ 2917us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:11:50.588523 (+ 1016us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:11:50.588701 (+ 178us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:11:50.594164 (+ 5463us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:11:50.594182 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:11:50.594221 (+ 39us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:11:50.594301 (+ 80us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:11:50.600306 (+ 6005us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:11:50.601768 (+ 1462us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:11:50.601786 (+ 18us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:11:50.601803 (+ 17us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:11:50.602147 (+ 344us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:11:50.602765 (+ 618us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:11:50.602773 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:11:50.602779 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:11:50.603829 (+ 1050us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:11:50.604301 (+ 472us) server_negotiation.cc:1035] Waiting for connection context
0928 02:11:50.606550 (+ 2249us) server_negotiation.cc:299] Negotiation successful
0928 02:11:50.607288 (+ 738us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11035,"thread_start_us":10893,"threads_started":1}
W20260928 02:11:50.882324 16083 sasl_common.cc:323] Time spent calling sasl_client_start(): real 1.440s user 0.001s sys 0.013s
W20260928 02:11:50.884760 16082 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:11:49.423160 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.235.65:35895 (local address 127.6.235.126:45241)
0928 02:11:49.423879 (+ 719us) server_negotiation.cc:206] Beginning negotiation
0928 02:11:49.423891 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0928 02:11:49.425722 (+ 1831us) server_negotiation.cc:407] Connection header received
0928 02:11:49.427840 (+ 2118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:11:49.427853 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:11:49.428209 (+ 356us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:11:49.428589 (+ 380us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:11:49.430173 (+ 1584us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:11:49.431341 (+ 1168us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:11:49.432960 (+ 1619us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:11:49.433621 (+ 661us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:11:49.433826 (+ 205us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:11:50.884205 (+1450379us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.235.65:35895: BlockingRecv error: recv got EOF from 127.6.235.65:35895 (error 108)
Metrics: {"server-negotiator.queue_time_us":462,"thread_start_us":291,"threads_started":1}
I20260928 02:11:50.887660 16083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:11:49.423274 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.235.126:45241 (local address 127.6.235.65:35895)
0928 02:11:49.425513 (+ 2239us) negotiation.cc:107] Waiting for socket to connect
0928 02:11:49.425578 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0928 02:11:49.427291 (+ 1713us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:11:49.428770 (+ 1479us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:11:49.428783 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:11:49.429416 (+ 633us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:11:49.429971 (+ 555us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:11:49.429989 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:11:49.431539 (+ 1550us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:11:49.431543 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:11:49.432789 (+ 1246us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:11:49.432800 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:11:49.433864 (+ 1064us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:11:49.433871 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:11:49.434143 (+ 272us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:11:49.442477 (+ 8334us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:11:49.442515 (+ 38us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:11:50.885859 (+1443344us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.235.126:45241: Server oryx/127.6.235.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":1952,"thread_start_us":1727,"threads_started":1}
W20260928 02:11:50.889060 16080 heartbeater.cc:644] Failed to heartbeat to 127.6.235.126:45241 (0 consecutive failures): Not authorized: Failed to ping master at 127.6.235.126:45241: Client connection negotiation failed: client connection to 127.6.235.126:45241: Server oryx/127.6.235.126@KRBTEST.COM not found in Kerberos database
Sep 28 02:11:50 dist-test-slave-86gv krb5kdc[15345](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.235.65@KRBTEST.COM for oryx/127.6.235.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 02:11:50 dist-test-slave-86gv krb5kdc[15345](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.235.65@KRBTEST.COM for oryx/127.6.235.126@KRBTEST.COM, Server not found in Kerberos database
W20260928 02:11:50.907912 16082 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:11:50.890160 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.235.65:36317 (local address 127.6.235.126:45241)
0928 02:11:50.890407 (+ 247us) server_negotiation.cc:206] Beginning negotiation
0928 02:11:50.890417 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 02:11:50.890570 (+ 153us) server_negotiation.cc:407] Connection header received
0928 02:11:50.890801 (+ 231us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:11:50.890809 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:11:50.890922 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:11:50.891087 (+ 165us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:11:50.892324 (+ 1237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:11:50.894028 (+ 1704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:11:50.897285 (+ 3257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:11:50.898280 (+ 995us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:11:50.898438 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:11:50.907519 (+ 9081us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.235.65:36317: BlockingRecv error: recv got EOF from 127.6.235.65:36317 (error 108)
Metrics: {"server-negotiator.queue_time_us":81}
I20260928 02:11:50.908987 16083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:11:50.890034 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.235.126:45241 (local address 127.6.235.65:36317)
0928 02:11:50.890435 (+ 401us) negotiation.cc:107] Waiting for socket to connect
0928 02:11:50.890464 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0928 02:11:50.890642 (+ 178us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:11:50.891146 (+ 504us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:11:50.891154 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:11:50.891846 (+ 692us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:11:50.892092 (+ 246us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:11:50.892110 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:11:50.894262 (+ 2152us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:11:50.894272 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:11:50.897007 (+ 2735us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:11:50.897034 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:11:50.898838 (+ 1804us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:11:50.898844 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:11:50.899135 (+ 291us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:11:50.900173 (+ 1038us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:11:50.900209 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:11:50.908530 (+ 8321us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.235.126:45241: Server oryx/127.6.235.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":203}
Sep 28 02:11:50 dist-test-slave-86gv krb5kdc[15345](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.235.65@KRBTEST.COM for oryx/127.6.235.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 02:11:50 dist-test-slave-86gv krb5kdc[15345](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.235.65@KRBTEST.COM for oryx/127.6.235.126@KRBTEST.COM, Server not found in Kerberos database
I20260928 02:11:50.928375 16083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:11:50.910482 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.235.126:45241 (local address 127.6.235.65:48629)
0928 02:11:50.911462 (+ 980us) negotiation.cc:107] Waiting for socket to connect
0928 02:11:50.911490 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0928 02:11:50.911732 (+ 242us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:11:50.912504 (+ 772us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:11:50.912511 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:11:50.913206 (+ 695us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:11:50.913533 (+ 327us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:11:50.913550 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:11:50.915750 (+ 2200us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:11:50.915759 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:11:50.918199 (+ 2440us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:11:50.918222 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:11:50.919871 (+ 1649us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:11:50.919880 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:11:50.920196 (+ 316us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:11:50.921225 (+ 1029us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:11:50.921260 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:11:50.927928 (+ 6668us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.235.126:45241: Server oryx/127.6.235.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":812}
W20260928 02:11:50.928745 16082 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:11:50.910977 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.235.65:48629 (local address 127.6.235.126:45241)
0928 02:11:50.911249 (+ 272us) server_negotiation.cc:206] Beginning negotiation
0928 02:11:50.911259 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 02:11:50.911591 (+ 332us) server_negotiation.cc:407] Connection header received
0928 02:11:50.911933 (+ 342us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:11:50.911943 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:11:50.912052 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:11:50.912180 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:11:50.913748 (+ 1568us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:11:50.915513 (+ 1765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:11:50.918529 (+ 3016us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:11:50.919658 (+ 1129us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:11:50.919840 (+ 182us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:11:50.928280 (+ 8440us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.235.65:48629: BlockingRecv error: recv got EOF from 127.6.235.65:48629 (error 108)
Metrics: {"server-negotiator.queue_time_us":89}
W20260928 02:11:50.929203 16080 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260928 02:11:51.310626 16216 sasl_common.cc:323] Time spent calling sasl_client_start(): real 1.338s user 0.005s sys 0.012s
W20260928 02:11:51.312366 16215 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:11:49.946572 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.235.66:39829 (local address 127.6.235.126:45241)
0928 02:11:49.956143 (+ 9571us) server_negotiation.cc:206] Beginning negotiation
0928 02:11:49.956164 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0928 02:11:49.959176 (+ 3012us) server_negotiation.cc:407] Connection header received
0928 02:11:49.962385 (+ 3209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:11:49.962395 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:11:49.962523 (+ 128us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:11:49.962701 (+ 178us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:11:49.964977 (+ 2276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:11:49.966702 (+ 1725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:11:49.969529 (+ 2827us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:11:49.970619 (+ 1090us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:11:49.970834 (+ 215us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:11:51.312069 (+1341235us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.235.66:39829: BlockingRecv error: recv got EOF from 127.6.235.66:39829 (error 108)
Metrics: {"server-negotiator.queue_time_us":9346,"thread_start_us":9166,"threads_started":1}
I20260928 02:11:51.313833 16216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:11:49.947422 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.235.126:45241 (local address 127.6.235.66:39829)
0928 02:11:49.958943 (+ 11521us) negotiation.cc:107] Waiting for socket to connect
0928 02:11:49.959010 (+ 67us) client_negotiation.cc:174] Beginning negotiation
0928 02:11:49.962012 (+ 3002us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:11:49.962991 (+ 979us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:11:49.963006 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:11:49.963815 (+ 809us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:11:49.964725 (+ 910us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:11:49.964753 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:11:49.966945 (+ 2192us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:11:49.966953 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:11:49.969333 (+ 2380us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:11:49.969352 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:11:49.970930 (+ 1578us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:11:49.970940 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:11:49.971385 (+ 445us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:11:49.972436 (+ 1051us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:11:49.972482 (+ 46us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:11:51.311925 (+1339443us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.235.126:45241: Server oryx/127.6.235.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":11124,"thread_start_us":11673,"threads_started":1}
W20260928 02:11:51.314782 16213 heartbeater.cc:644] Failed to heartbeat to 127.6.235.126:45241 (0 consecutive failures): Not authorized: Failed to ping master at 127.6.235.126:45241: Client connection negotiation failed: client connection to 127.6.235.126:45241: Server oryx/127.6.235.126@KRBTEST.COM not found in Kerberos database
Sep 28 02:11:51 dist-test-slave-86gv krb5kdc[15345](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.235.66@KRBTEST.COM for oryx/127.6.235.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 02:11:51 dist-test-slave-86gv krb5kdc[15345](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.235.66@KRBTEST.COM for oryx/127.6.235.126@KRBTEST.COM, Server not found in Kerberos database
W20260928 02:11:51.328778 16215 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:11:51.315533 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.235.66:47627 (local address 127.6.235.126:45241)
0928 02:11:51.315685 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0928 02:11:51.315689 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:11:51.315775 (+ 86us) server_negotiation.cc:407] Connection header received
0928 02:11:51.316059 (+ 284us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:11:51.316063 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:11:51.316125 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:11:51.316206 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:11:51.317594 (+ 1388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:11:51.318343 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:11:51.321108 (+ 2765us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:11:51.321521 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:11:51.321624 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:11:51.328605 (+ 6981us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.235.66:47627: BlockingRecv error: recv got EOF from 127.6.235.66:47627 (error 108)
Metrics: {"server-negotiator.queue_time_us":42}
I20260928 02:11:51.328872 16216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:11:51.315392 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.235.126:45241 (local address 127.6.235.66:47627)
0928 02:11:51.315660 (+ 268us) negotiation.cc:107] Waiting for socket to connect
0928 02:11:51.315690 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0928 02:11:51.315869 (+ 179us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:11:51.316319 (+ 450us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:11:51.316326 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:11:51.317048 (+ 722us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:11:51.317388 (+ 340us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:11:51.317406 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:11:51.318544 (+ 1138us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:11:51.318553 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:11:51.320924 (+ 2371us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:11:51.320949 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:11:51.321740 (+ 791us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:11:51.321749 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:11:51.322013 (+ 264us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:11:51.322993 (+ 980us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:11:51.323029 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:11:51.328438 (+ 5409us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.235.126:45241: Server oryx/127.6.235.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":74}
Sep 28 02:11:51 dist-test-slave-86gv krb5kdc[15345](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.235.66@KRBTEST.COM for oryx/127.6.235.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 02:11:51 dist-test-slave-86gv krb5kdc[15345](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.235.66@KRBTEST.COM for oryx/127.6.235.126@KRBTEST.COM, Server not found in Kerberos database
I20260928 02:11:51.341714 16216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:11:51.329702 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.235.126:45241 (local address 127.6.235.66:36047)
0928 02:11:51.329865 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0928 02:11:51.329884 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0928 02:11:51.329991 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:11:51.330641 (+ 650us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:11:51.330644 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:11:51.331030 (+ 386us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:11:51.331234 (+ 204us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:11:51.331244 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:11:51.333277 (+ 2033us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:11:51.333281 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:11:51.334235 (+ 954us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:11:51.334248 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:11:51.335623 (+ 1375us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:11:51.335631 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:11:51.335835 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:11:51.336567 (+ 732us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:11:51.336596 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:11:51.341511 (+ 4915us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.235.126:45241: Server oryx/127.6.235.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":57}
W20260928 02:11:51.342258 16215 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:11:51.329989 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.235.66:36047 (local address 127.6.235.126:45241)
0928 02:11:51.330224 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0928 02:11:51.330234 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 02:11:51.330260 (+ 26us) server_negotiation.cc:407] Connection header received
0928 02:11:51.330351 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:11:51.330360 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:11:51.330466 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:11:51.330612 (+ 146us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:11:51.331394 (+ 782us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:11:51.333091 (+ 1697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:11:51.334398 (+ 1307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:11:51.335424 (+ 1026us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:11:51.335607 (+ 183us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:11:51.341872 (+ 6265us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.235.66:36047: BlockingRecv error: recv got EOF from 127.6.235.66:36047 (error 108)
Metrics: {"server-negotiator.queue_time_us":70}
W20260928 02:11:51.342356 16213 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260928 02:11:51.766500 16349 sasl_common.cc:323] Time spent calling sasl_client_start(): real 1.186s user 0.010s sys 0.015s
W20260928 02:11:51.767863 16348 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:11:50.554084 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.235.67:49875 (local address 127.6.235.126:45241)
0928 02:11:50.566423 (+ 12339us) server_negotiation.cc:206] Beginning negotiation
0928 02:11:50.566433 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 02:11:50.566462 (+ 29us) server_negotiation.cc:407] Connection header received
0928 02:11:50.566553 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:11:50.566561 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:11:50.566677 (+ 116us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:11:50.570253 (+ 3576us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:11:50.570648 (+ 395us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:11:50.572529 (+ 1881us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:11:50.576396 (+ 3867us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:11:50.577519 (+ 1123us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:11:50.578965 (+ 1446us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:11:51.767635 (+1188670us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.235.67:49875: BlockingRecv error: recv got EOF from 127.6.235.67:49875 (error 108)
Metrics: {"server-negotiator.queue_time_us":12090,"thread_start_us":11882,"threads_started":1}
I20260928 02:11:51.768481 16349 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:11:50.555231 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.235.126:45241 (local address 127.6.235.67:49875)
0928 02:11:50.561548 (+ 6317us) negotiation.cc:107] Waiting for socket to connect
0928 02:11:50.561603 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0928 02:11:50.565199 (+ 3596us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:11:50.567396 (+ 2197us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:11:50.567416 (+ 20us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:11:50.568667 (+ 1251us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:11:50.569988 (+ 1321us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:11:50.570031 (+ 43us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:11:50.572852 (+ 2821us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:11:50.572861 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:11:50.576061 (+ 3200us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:11:50.576088 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:11:50.577887 (+ 1799us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:11:50.577897 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:11:50.578362 (+ 465us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:11:50.580051 (+ 1689us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:11:50.580097 (+ 46us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:11:51.767387 (+1187290us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.235.126:45241: Server oryx/127.6.235.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":5962,"thread_start_us":5681,"threads_started":1}
W20260928 02:11:51.769451 16346 heartbeater.cc:644] Failed to heartbeat to 127.6.235.126:45241 (0 consecutive failures): Not authorized: Failed to ping master at 127.6.235.126:45241: Client connection negotiation failed: client connection to 127.6.235.126:45241: Server oryx/127.6.235.126@KRBTEST.COM not found in Kerberos database
Sep 28 02:11:51 dist-test-slave-86gv krb5kdc[15345](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.235.67@KRBTEST.COM for oryx/127.6.235.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 02:11:51 dist-test-slave-86gv krb5kdc[15345](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.235.67@KRBTEST.COM for oryx/127.6.235.126@KRBTEST.COM, Server not found in Kerberos database
I20260928 02:11:51.779228 16349 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:11:51.770148 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.235.126:45241 (local address 127.6.235.67:60817)
0928 02:11:51.770366 (+ 218us) negotiation.cc:107] Waiting for socket to connect
0928 02:11:51.770388 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0928 02:11:51.770505 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:11:51.770816 (+ 311us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:11:51.770822 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:11:51.771165 (+ 343us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:11:51.771331 (+ 166us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:11:51.771340 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:11:51.772479 (+ 1139us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:11:51.772484 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:11:51.773571 (+ 1087us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:11:51.773584 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:11:51.774259 (+ 675us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:11:51.774264 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:11:51.774398 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:11:51.775047 (+ 649us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:11:51.775069 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:11:51.779014 (+ 3945us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.235.126:45241: Server oryx/127.6.235.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":66}
W20260928 02:11:51.779577 16348 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:11:51.770310 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.235.67:60817 (local address 127.6.235.126:45241)
0928 02:11:51.770501 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0928 02:11:51.770505 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:11:51.770523 (+ 18us) server_negotiation.cc:407] Connection header received
0928 02:11:51.770595 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:11:51.770598 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:11:51.770654 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:11:51.770736 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:11:51.771530 (+ 794us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:11:51.772344 (+ 814us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:11:51.773703 (+ 1359us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:11:51.774099 (+ 396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:11:51.774195 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:11:51.779212 (+ 5017us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.235.67:60817: BlockingRecv error: recv got EOF from 127.6.235.67:60817 (error 108)
Metrics: {"server-negotiator.queue_time_us":87}
Sep 28 02:11:51 dist-test-slave-86gv krb5kdc[15345](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.235.67@KRBTEST.COM for oryx/127.6.235.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 02:11:51 dist-test-slave-86gv krb5kdc[15345](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.235.67@KRBTEST.COM for oryx/127.6.235.126@KRBTEST.COM, Server not found in Kerberos database
I20260928 02:11:51.788437 16349 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:11:51.779918 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.235.126:45241 (local address 127.6.235.67:44737)
0928 02:11:51.780080 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0928 02:11:51.780103 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0928 02:11:51.780205 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:11:51.780556 (+ 351us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:11:51.780560 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:11:51.780861 (+ 301us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:11:51.781075 (+ 214us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:11:51.781084 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:11:51.782052 (+ 968us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:11:51.782059 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:11:51.782979 (+ 920us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:11:51.782989 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:11:51.783697 (+ 708us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:11:51.783702 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:11:51.783882 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:11:51.784478 (+ 596us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:11:51.784494 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:11:51.788153 (+ 3659us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.235.126:45241: Server oryx/127.6.235.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":52}
W20260928 02:11:51.788614 16348 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:11:51.779966 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.235.67:44737 (local address 127.6.235.126:45241)
0928 02:11:51.780215 (+ 249us) server_negotiation.cc:206] Beginning negotiation
0928 02:11:51.780220 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 02:11:51.780238 (+ 18us) server_negotiation.cc:407] Connection header received
0928 02:11:51.780299 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:11:51.780303 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:11:51.780385 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:11:51.780487 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:11:51.781204 (+ 717us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:11:51.781909 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:11:51.783115 (+ 1206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:11:51.783535 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:11:51.783682 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:11:51.788347 (+ 4665us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.235.67:44737: BlockingRecv error: recv got EOF from 127.6.235.67:44737 (error 108)
Metrics: {"server-negotiator.queue_time_us":142}
W20260928 02:11:51.788803 16346 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Sep 28 02:11:51 dist-test-slave-86gv krb5kdc[15345](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.235.65@KRBTEST.COM for oryx/127.6.235.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 02:11:51 dist-test-slave-86gv krb5kdc[15345](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.235.65@KRBTEST.COM for oryx/127.6.235.126@KRBTEST.COM, Server not found in Kerberos database
I20260928 02:11:51.940043 16353 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:11:51.929873 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.235.126:45241 (local address 127.6.235.65:54817)
0928 02:11:51.930410 (+ 537us) negotiation.cc:107] Waiting for socket to connect
0928 02:11:51.930425 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 02:11:51.930616 (+ 191us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:11:51.930982 (+ 366us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:11:51.930986 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:11:51.931466 (+ 480us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:11:51.931615 (+ 149us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:11:51.931624 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:11:51.932609 (+ 985us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:11:51.932616 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:11:51.933770 (+ 1154us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:11:51.933785 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:11:51.934883 (+ 1098us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:11:51.934891 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:11:51.935118 (+ 227us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:11:51.935959 (+ 841us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:11:51.935981 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:11:51.939783 (+ 3802us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.235.126:45241: Server oryx/127.6.235.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":434,"thread_start_us":308,"threads_started":1}
W20260928 02:11:51.940397 16348 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:11:51.930063 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.235.65:54817 (local address 127.6.235.126:45241)
0928 02:11:51.930305 (+ 242us) server_negotiation.cc:206] Beginning negotiation
0928 02:11:51.930311 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 02:11:51.930514 (+ 203us) server_negotiation.cc:407] Connection header received
0928 02:11:51.930751 (+ 237us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:11:51.930765 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:11:51.930843 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:11:51.930952 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:11:51.931763 (+ 811us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:11:51.932453 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:11:51.933979 (+ 1526us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:11:51.934667 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:11:51.934797 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:11:51.940060 (+ 5263us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.235.65:54817: BlockingRecv error: recv got EOF from 127.6.235.65:54817 (error 108)
Metrics: {"server-negotiator.queue_time_us":92}
Sep 28 02:11:52 dist-test-slave-86gv krb5kdc[15345](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.235.66@KRBTEST.COM for oryx/127.6.235.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 02:11:52 dist-test-slave-86gv krb5kdc[15345](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.235.66@KRBTEST.COM for oryx/127.6.235.126@KRBTEST.COM, Server not found in Kerberos database
I20260928 02:11:52.352728 16354 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:11:52.343049 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.235.126:45241 (local address 127.6.235.66:58435)
0928 02:11:52.343752 (+ 703us) negotiation.cc:107] Waiting for socket to connect
0928 02:11:52.343774 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0928 02:11:52.343927 (+ 153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:11:52.344303 (+ 376us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:11:52.344308 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:11:52.344893 (+ 585us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:11:52.345117 (+ 224us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:11:52.345152 (+ 35us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:11:52.346442 (+ 1290us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:11:52.346445 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:11:52.347368 (+ 923us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:11:52.347380 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:11:52.348201 (+ 821us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:11:52.348206 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:11:52.348319 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:11:52.348988 (+ 669us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:11:52.349006 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:11:52.352478 (+ 3472us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.235.126:45241: Server oryx/127.6.235.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":567,"thread_start_us":455,"threads_started":1}
W20260928 02:11:52.352957 16348 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:11:52.343203 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.235.66:58435 (local address 127.6.235.126:45241)
0928 02:11:52.343389 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0928 02:11:52.343393 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:11:52.343891 (+ 498us) server_negotiation.cc:407] Connection header received
0928 02:11:52.344050 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:11:52.344056 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:11:52.344145 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:11:52.344230 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:11:52.345287 (+ 1057us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:11:52.346303 (+ 1016us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:11:52.347528 (+ 1225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:11:52.348037 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:11:52.348147 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:11:52.352690 (+ 4543us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.235.66:58435: BlockingRecv error: recv got EOF from 127.6.235.66:58435 (error 108)
Metrics: {"server-negotiator.queue_time_us":70}
Sep 28 02:11:52 dist-test-slave-86gv krb5kdc[15345](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.235.67@KRBTEST.COM for oryx/127.6.235.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 02:11:52 dist-test-slave-86gv krb5kdc[15345](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.235.67@KRBTEST.COM for oryx/127.6.235.126@KRBTEST.COM, Server not found in Kerberos database
I20260928 02:11:52.799170 16355 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:11:52.789760 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.235.126:45241 (local address 127.6.235.67:36937)
0928 02:11:52.790326 (+ 566us) negotiation.cc:107] Waiting for socket to connect
0928 02:11:52.790342 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 02:11:52.790505 (+ 163us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:11:52.790864 (+ 359us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:11:52.790869 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:11:52.791304 (+ 435us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:11:52.791444 (+ 140us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:11:52.791453 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:11:52.792473 (+ 1020us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:11:52.792479 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:11:52.793460 (+ 981us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:11:52.793473 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:11:52.794259 (+ 786us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:11:52.794267 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:11:52.794437 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:11:52.795111 (+ 674us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:11:52.795129 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:11:52.798766 (+ 3637us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.235.126:45241: Server oryx/127.6.235.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":455,"thread_start_us":337,"threads_started":1}
W20260928 02:11:52.799185 16348 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:11:52.789856 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.235.67:36937 (local address 127.6.235.126:45241)
0928 02:11:52.790041 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0928 02:11:52.790047 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 02:11:52.790438 (+ 391us) server_negotiation.cc:407] Connection header received
0928 02:11:52.790634 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:11:52.790639 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:11:52.790725 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:11:52.790831 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:11:52.791598 (+ 767us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:11:52.792323 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:11:52.793627 (+ 1304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:11:52.794064 (+ 437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:11:52.794194 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:11:52.798980 (+ 4786us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.235.67:36937: BlockingRecv error: recv got EOF from 127.6.235.67:36937 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
Sep 28 02:11:52 dist-test-slave-86gv krb5kdc[15345](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.235.65@KRBTEST.COM for oryx/127.6.235.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 02:11:52 dist-test-slave-86gv krb5kdc[15345](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.235.65@KRBTEST.COM for oryx/127.6.235.126@KRBTEST.COM, Server not found in Kerberos database
I20260928 02:11:52.950695 16356 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:11:52.940993 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.235.126:45241 (local address 127.6.235.65:42769)
0928 02:11:52.941498 (+ 505us) negotiation.cc:107] Waiting for socket to connect
0928 02:11:52.941519 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0928 02:11:52.941670 (+ 151us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:11:52.941989 (+ 319us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:11:52.941992 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:11:52.942465 (+ 473us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:11:52.942644 (+ 179us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:11:52.942653 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:11:52.943670 (+ 1017us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:11:52.943673 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:11:52.944584 (+ 911us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:11:52.944595 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:11:52.945373 (+ 778us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:11:52.945377 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:11:52.945526 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:11:52.946197 (+ 671us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:11:52.946218 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:11:52.950337 (+ 4119us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.235.126:45241: Server oryx/127.6.235.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":385,"thread_start_us":283,"threads_started":1}
W20260928 02:11:52.951025 16348 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:11:52.941071 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.235.65:42769 (local address 127.6.235.126:45241)
0928 02:11:52.941264 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0928 02:11:52.941272 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 02:11:52.941621 (+ 349us) server_negotiation.cc:407] Connection header received
0928 02:11:52.941799 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:11:52.941803 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:11:52.941864 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:11:52.941950 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:11:52.942780 (+ 830us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:11:52.943534 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:11:52.944710 (+ 1176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:11:52.945220 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:11:52.945317 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:11:52.950629 (+ 5312us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.235.65:42769: BlockingRecv error: recv got EOF from 127.6.235.65:42769 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
Sep 28 02:11:53 dist-test-slave-86gv krb5kdc[15345](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.235.66@KRBTEST.COM for oryx/127.6.235.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 02:11:53 dist-test-slave-86gv krb5kdc[15345](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.235.66@KRBTEST.COM for oryx/127.6.235.126@KRBTEST.COM, Server not found in Kerberos database
I20260928 02:11:53.363407 16357 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:11:53.353856 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.235.126:45241 (local address 127.6.235.66:35845)
0928 02:11:53.354471 (+ 615us) negotiation.cc:107] Waiting for socket to connect
0928 02:11:53.354487 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 02:11:53.354627 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:11:53.355056 (+ 429us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:11:53.355061 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:11:53.355552 (+ 491us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:11:53.355741 (+ 189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:11:53.355750 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:11:53.356953 (+ 1203us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:11:53.356960 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:11:53.357907 (+ 947us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:11:53.357918 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:11:53.358634 (+ 716us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:11:53.358640 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:11:53.358815 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:11:53.359394 (+ 579us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:11:53.359411 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:11:53.363052 (+ 3641us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.235.126:45241: Server oryx/127.6.235.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":525,"thread_start_us":416,"threads_started":1}
W20260928 02:11:53.363492 16348 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:11:53.353955 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.235.66:35845 (local address 127.6.235.126:45241)
0928 02:11:53.354143 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0928 02:11:53.354147 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:11:53.354577 (+ 430us) server_negotiation.cc:407] Connection header received
0928 02:11:53.354793 (+ 216us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:11:53.354798 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:11:53.354868 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:11:53.354975 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:11:53.355887 (+ 912us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:11:53.356787 (+ 900us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:11:53.358062 (+ 1275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:11:53.358472 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:11:53.358619 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:11:53.363257 (+ 4638us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.235.66:35845: BlockingRecv error: recv got EOF from 127.6.235.66:35845 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
Sep 28 02:11:53 dist-test-slave-86gv krb5kdc[15345](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.235.67@KRBTEST.COM for oryx/127.6.235.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 02:11:53 dist-test-slave-86gv krb5kdc[15345](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.235.67@KRBTEST.COM for oryx/127.6.235.126@KRBTEST.COM, Server not found in Kerberos database
I20260928 02:11:53.809466 16358 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:11:53.800136 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.235.126:45241 (local address 127.6.235.67:58769)
0928 02:11:53.800714 (+ 578us) negotiation.cc:107] Waiting for socket to connect
0928 02:11:53.800730 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 02:11:53.800845 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:11:53.801197 (+ 352us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:11:53.801200 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:11:53.801621 (+ 421us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:11:53.801798 (+ 177us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:11:53.801806 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:11:53.802852 (+ 1046us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:11:53.802856 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:11:53.803770 (+ 914us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:11:53.803782 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:11:53.804484 (+ 702us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:11:53.804527 (+ 43us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:11:53.804653 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:11:53.805418 (+ 765us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:11:53.805435 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:11:53.809120 (+ 3685us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.235.126:45241: Server oryx/127.6.235.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":482,"thread_start_us":378,"threads_started":1}
W20260928 02:11:53.809576 16348 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:11:53.800405 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.235.67:58769 (local address 127.6.235.126:45241)
0928 02:11:53.800654 (+ 249us) server_negotiation.cc:206] Beginning negotiation
0928 02:11:53.800660 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 02:11:53.800805 (+ 145us) server_negotiation.cc:407] Connection header received
0928 02:11:53.800967 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:11:53.800971 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:11:53.801057 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:11:53.801140 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:11:53.801932 (+ 792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:11:53.802719 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:11:53.803889 (+ 1170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:11:53.804349 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:11:53.804441 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:11:53.809303 (+ 4862us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.235.67:58769: BlockingRecv error: recv got EOF from 127.6.235.67:58769 (error 108)
Metrics: {"server-negotiator.queue_time_us":122}
Sep 28 02:11:53 dist-test-slave-86gv krb5kdc[15345](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.235.65@KRBTEST.COM for oryx/127.6.235.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 02:11:53 dist-test-slave-86gv krb5kdc[15345](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.235.65@KRBTEST.COM for oryx/127.6.235.126@KRBTEST.COM, Server not found in Kerberos database
I20260928 02:11:53.962518 16359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:11:53.951644 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.235.126:45241 (local address 127.6.235.65:57559)
0928 02:11:53.952251 (+ 607us) negotiation.cc:107] Waiting for socket to connect
0928 02:11:53.952273 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0928 02:11:53.952485 (+ 212us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:11:53.953002 (+ 517us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:11:53.953010 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:11:53.953534 (+ 524us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:11:53.953817 (+ 283us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:11:53.953831 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:11:53.954886 (+ 1055us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:11:53.954895 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:11:53.956681 (+ 1786us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:11:53.956700 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:11:53.957598 (+ 898us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:11:53.957605 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:11:53.957777 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:11:53.958650 (+ 873us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:11:53.958669 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:11:53.962164 (+ 3495us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.235.126:45241: Server oryx/127.6.235.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":442,"thread_start_us":274,"threads_started":1}
W20260928 02:11:53.962612 16348 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:11:53.951818 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.235.65:57559 (local address 127.6.235.126:45241)
0928 02:11:53.952055 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0928 02:11:53.952062 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 02:11:53.952378 (+ 316us) server_negotiation.cc:407] Connection header received
0928 02:11:53.952695 (+ 317us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:11:53.952699 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:11:53.952761 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:11:53.952846 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:11:53.953988 (+ 1142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:11:53.954737 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:11:53.956836 (+ 2099us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:11:53.957428 (+ 592us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:11:53.957583 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:11:53.962385 (+ 4802us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.235.65:57559: BlockingRecv error: recv got EOF from 127.6.235.65:57559 (error 108)
Metrics: {"server-negotiator.queue_time_us":95}
Sep 28 02:11:54 dist-test-slave-86gv krb5kdc[15345](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.235.66@KRBTEST.COM for oryx/127.6.235.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 02:11:54 dist-test-slave-86gv krb5kdc[15345](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.235.66@KRBTEST.COM for oryx/127.6.235.126@KRBTEST.COM, Server not found in Kerberos database
I20260928 02:11:54.380379 16360 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:11:54.364424 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.235.126:45241 (local address 127.6.235.66:59071)
0928 02:11:54.364959 (+ 535us) negotiation.cc:107] Waiting for socket to connect
0928 02:11:54.364977 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0928 02:11:54.365152 (+ 175us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:11:54.365525 (+ 373us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:11:54.365532 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:11:54.365957 (+ 425us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:11:54.366167 (+ 210us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:11:54.366177 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:11:54.367222 (+ 1045us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:11:54.367226 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:11:54.368145 (+ 919us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:11:54.368156 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:11:54.368841 (+ 685us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:11:54.368845 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:11:54.369037 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:11:54.376118 (+ 7081us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:11:54.376140 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:11:54.380144 (+ 4004us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.235.126:45241: Server oryx/127.6.235.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":375,"thread_start_us":260,"threads_started":1}
W20260928 02:11:54.380595 16348 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:11:54.364571 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.235.66:59071 (local address 127.6.235.126:45241)
0928 02:11:54.364814 (+ 243us) server_negotiation.cc:206] Beginning negotiation
0928 02:11:54.364820 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 02:11:54.365066 (+ 246us) server_negotiation.cc:407] Connection header received
0928 02:11:54.365291 (+ 225us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:11:54.365295 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:11:54.365358 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:11:54.365464 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:11:54.366326 (+ 862us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:11:54.367089 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:11:54.368277 (+ 1188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:11:54.368711 (+ 434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:11:54.368809 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:11:54.380356 (+ 11547us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.235.66:59071: BlockingRecv error: recv got EOF from 127.6.235.66:59071 (error 108)
Metrics: {"server-negotiator.queue_time_us":95}
Sep 28 02:11:54 dist-test-slave-86gv krb5kdc[15345](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.235.67@KRBTEST.COM for oryx/127.6.235.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 02:11:54 dist-test-slave-86gv krb5kdc[15345](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.235.67@KRBTEST.COM for oryx/127.6.235.126@KRBTEST.COM, Server not found in Kerberos database
I20260928 02:11:54.819046 16361 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:11:54.810450 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.235.126:45241 (local address 127.6.235.67:37619)
0928 02:11:54.810956 (+ 506us) negotiation.cc:107] Waiting for socket to connect
0928 02:11:54.810972 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 02:11:54.811149 (+ 177us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:11:54.811440 (+ 291us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:11:54.811444 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:11:54.811808 (+ 364us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:11:54.811965 (+ 157us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:11:54.811974 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:11:54.813126 (+ 1152us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:11:54.813130 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:11:54.814012 (+ 882us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:11:54.814023 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:11:54.814692 (+ 669us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:11:54.814696 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:11:54.814820 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:11:54.815425 (+ 605us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:11:54.815441 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:11:54.818781 (+ 3340us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.235.126:45241: Server oryx/127.6.235.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":400,"thread_start_us":298,"threads_started":1}
W20260928 02:11:54.819183 16348 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:11:54.810530 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.235.67:37619 (local address 127.6.235.126:45241)
0928 02:11:54.810822 (+ 292us) server_negotiation.cc:206] Beginning negotiation
0928 02:11:54.810836 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0928 02:11:54.811106 (+ 270us) server_negotiation.cc:407] Connection header received
0928 02:11:54.811246 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:11:54.811249 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:11:54.811320 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:11:54.811410 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:11:54.812146 (+ 736us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:11:54.813003 (+ 857us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:11:54.814141 (+ 1138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:11:54.814572 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:11:54.814669 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:11:54.818980 (+ 4311us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.235.67:37619: BlockingRecv error: recv got EOF from 127.6.235.67:37619 (error 108)
Metrics: {"server-negotiator.queue_time_us":107}
Sep 28 02:11:54 dist-test-slave-86gv krb5kdc[15345](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.235.65@KRBTEST.COM for oryx/127.6.235.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 02:11:54 dist-test-slave-86gv krb5kdc[15345](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.235.65@KRBTEST.COM for oryx/127.6.235.126@KRBTEST.COM, Server not found in Kerberos database
W20260928 02:11:54.973176 16348 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:11:54.963570 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.235.65:48643 (local address 127.6.235.126:45241)
0928 02:11:54.963742 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0928 02:11:54.963747 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 02:11:54.964077 (+ 330us) server_negotiation.cc:407] Connection header received
0928 02:11:54.964215 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:11:54.964219 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:11:54.964279 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:11:54.964398 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:11:54.965195 (+ 797us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:11:54.965985 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:11:54.967262 (+ 1277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:11:54.967702 (+ 440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:11:54.967808 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:11:54.972959 (+ 5151us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.235.65:48643: BlockingRecv error: recv got EOF from 127.6.235.65:48643 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
I20260928 02:11:54.973176 16362 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:11:54.963458 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.235.126:45241 (local address 127.6.235.65:48643)
0928 02:11:54.963956 (+ 498us) negotiation.cc:107] Waiting for socket to connect
0928 02:11:54.963972 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 02:11:54.964115 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:11:54.964430 (+ 315us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:11:54.964433 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:11:54.964809 (+ 376us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:11:54.965069 (+ 260us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:11:54.965080 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:11:54.966112 (+ 1032us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:11:54.966115 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:11:54.967119 (+ 1004us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:11:54.967130 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:11:54.967853 (+ 723us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:11:54.967858 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:11:54.968051 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:11:54.968856 (+ 805us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:11:54.968878 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:11:54.972720 (+ 3842us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.235.126:45241: Server oryx/127.6.235.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":386,"thread_start_us":280,"threads_started":1}
Sep 28 02:11:55 dist-test-slave-86gv krb5kdc[15345](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.235.66@KRBTEST.COM for oryx/127.6.235.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 02:11:55 dist-test-slave-86gv krb5kdc[15345](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.235.66@KRBTEST.COM for oryx/127.6.235.126@KRBTEST.COM, Server not found in Kerberos database
I20260928 02:11:55.391345 16363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:11:55.381545 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.235.126:45241 (local address 127.6.235.66:41103)
0928 02:11:55.382096 (+ 551us) negotiation.cc:107] Waiting for socket to connect
0928 02:11:55.382113 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0928 02:11:55.382267 (+ 154us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:11:55.382644 (+ 377us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:11:55.382650 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:11:55.383033 (+ 383us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:11:55.383192 (+ 159us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:11:55.383201 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:11:55.384204 (+ 1003us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:11:55.384211 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:11:55.385257 (+ 1046us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:11:55.385268 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:11:55.386305 (+ 1037us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:11:55.386310 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:11:55.386460 (+ 150us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:11:55.387226 (+ 766us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:11:55.387242 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:11:55.391001 (+ 3759us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.235.126:45241: Server oryx/127.6.235.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":458,"thread_start_us":295,"threads_started":1}
W20260928 02:11:55.393848 16348 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:11:55.381623 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.235.66:41103 (local address 127.6.235.126:45241)
0928 02:11:55.381824 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0928 02:11:55.381832 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 02:11:55.382207 (+ 375us) server_negotiation.cc:407] Connection header received
0928 02:11:55.382401 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:11:55.382405 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:11:55.382467 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:11:55.382595 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:11:55.383334 (+ 739us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:11:55.384075 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:11:55.385414 (+ 1339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:11:55.386167 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:11:55.386299 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:11:55.393602 (+ 7303us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.235.66:41103: BlockingRecv error: recv got EOF from 127.6.235.66:41103 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
Sep 28 02:11:55 dist-test-slave-86gv krb5kdc[15345](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.235.67@KRBTEST.COM for oryx/127.6.235.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 02:11:55 dist-test-slave-86gv krb5kdc[15345](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.235.67@KRBTEST.COM for oryx/127.6.235.126@KRBTEST.COM, Server not found in Kerberos database
I20260928 02:11:55.830104 16364 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:11:55.819965 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.235.126:45241 (local address 127.6.235.67:44931)
0928 02:11:55.820634 (+ 669us) negotiation.cc:107] Waiting for socket to connect
0928 02:11:55.820650 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 02:11:55.820801 (+ 151us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:11:55.821191 (+ 390us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:11:55.821196 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:11:55.821591 (+ 395us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:11:55.821758 (+ 167us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:11:55.821767 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:11:55.823051 (+ 1284us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:11:55.823060 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:11:55.824330 (+ 1270us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:11:55.824345 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:11:55.825181 (+ 836us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:11:55.825188 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:11:55.825393 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:11:55.826049 (+ 656us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:11:55.826068 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:11:55.829739 (+ 3671us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.235.126:45241: Server oryx/127.6.235.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":545,"thread_start_us":398,"threads_started":1}
W20260928 02:11:55.830297 16348 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:11:55.820281 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.235.67:44931 (local address 127.6.235.126:45241)
0928 02:11:55.820557 (+ 276us) server_negotiation.cc:206] Beginning negotiation
0928 02:11:55.820563 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 02:11:55.820731 (+ 168us) server_negotiation.cc:407] Connection header received
0928 02:11:55.820948 (+ 217us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:11:55.820953 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:11:55.821053 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:11:55.821153 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:11:55.821896 (+ 743us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:11:55.822781 (+ 885us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:11:55.824490 (+ 1709us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:11:55.825020 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:11:55.825165 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:11:55.829990 (+ 4825us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.235.67:44931: BlockingRecv error: recv got EOF from 127.6.235.67:44931 (error 108)
Metrics: {"server-negotiator.queue_time_us":85}
Sep 28 02:11:55 dist-test-slave-86gv krb5kdc[15345](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.235.65@KRBTEST.COM for oryx/127.6.235.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 02:11:55 dist-test-slave-86gv krb5kdc[15345](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.235.65@KRBTEST.COM for oryx/127.6.235.126@KRBTEST.COM, Server not found in Kerberos database
I20260928 02:11:55.983225 16365 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:11:55.974150 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.235.126:45241 (local address 127.6.235.65:45975)
0928 02:11:55.974592 (+ 442us) negotiation.cc:107] Waiting for socket to connect
0928 02:11:55.974608 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 02:11:55.974726 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:11:55.975020 (+ 294us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:11:55.975023 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:11:55.975398 (+ 375us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:11:55.975555 (+ 157us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:11:55.975563 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:11:55.976716 (+ 1153us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:11:55.976722 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:11:55.978101 (+ 1379us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:11:55.978117 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:11:55.978785 (+ 668us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:11:55.978789 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:11:55.978904 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:11:55.979522 (+ 618us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:11:55.979538 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:11:55.982995 (+ 3457us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.235.126:45241: Server oryx/127.6.235.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":335,"thread_start_us":239,"threads_started":1}
W20260928 02:11:55.983441 16348 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:11:55.974245 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.235.65:45975 (local address 127.6.235.126:45241)
0928 02:11:55.974500 (+ 255us) server_negotiation.cc:206] Beginning negotiation
0928 02:11:55.974507 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 02:11:55.974691 (+ 184us) server_negotiation.cc:407] Connection header received
0928 02:11:55.974825 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:11:55.974829 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:11:55.974890 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:11:55.974975 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:11:55.975676 (+ 701us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:11:55.976576 (+ 900us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:11:55.978255 (+ 1679us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:11:55.978667 (+ 412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:11:55.978767 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:11:55.983191 (+ 4424us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.235.65:45975: BlockingRecv error: recv got EOF from 127.6.235.65:45975 (error 108)
Metrics: {"server-negotiator.queue_time_us":112}
Sep 28 02:11:56 dist-test-slave-86gv krb5kdc[15345](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.235.66@KRBTEST.COM for oryx/127.6.235.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 02:11:56 dist-test-slave-86gv krb5kdc[15345](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.235.66@KRBTEST.COM for oryx/127.6.235.126@KRBTEST.COM, Server not found in Kerberos database
I20260928 02:11:56.403656 16366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:11:56.394531 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.235.126:45241 (local address 127.6.235.66:42605)
0928 02:11:56.395294 (+ 763us) negotiation.cc:107] Waiting for socket to connect
0928 02:11:56.395310 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 02:11:56.395466 (+ 156us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:11:56.395789 (+ 323us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:11:56.395793 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:11:56.396318 (+ 525us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:11:56.396463 (+ 145us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:11:56.396481 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:11:56.397593 (+ 1112us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:11:56.397600 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:11:56.398519 (+ 919us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:11:56.398531 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:11:56.399230 (+ 699us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:11:56.399236 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:11:56.399412 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:11:56.399980 (+ 568us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:11:56.399997 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:11:56.403319 (+ 3322us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.235.126:45241: Server oryx/127.6.235.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":642,"thread_start_us":340,"threads_started":1}
W20260928 02:11:56.404038 16348 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:11:56.394752 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.235.66:42605 (local address 127.6.235.126:45241)
0928 02:11:56.394942 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0928 02:11:56.394948 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 02:11:56.395396 (+ 448us) server_negotiation.cc:407] Connection header received
0928 02:11:56.395591 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:11:56.395596 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:11:56.395664 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:11:56.395763 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:11:56.396607 (+ 844us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:11:56.397460 (+ 853us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:11:56.398661 (+ 1201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:11:56.399079 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:11:56.399208 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:11:56.403542 (+ 4334us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.235.66:42605: BlockingRecv error: recv got EOF from 127.6.235.66:42605 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
Sep 28 02:11:56 dist-test-slave-86gv krb5kdc[15345](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.235.67@KRBTEST.COM for oryx/127.6.235.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 02:11:56 dist-test-slave-86gv krb5kdc[15345](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.235.67@KRBTEST.COM for oryx/127.6.235.126@KRBTEST.COM, Server not found in Kerberos database
I20260928 02:11:56.840296 16367 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:11:56.831342 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.235.126:45241 (local address 127.6.235.67:51881)
0928 02:11:56.831803 (+ 461us) negotiation.cc:107] Waiting for socket to connect
0928 02:11:56.831819 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 02:11:56.831934 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:11:56.832268 (+ 334us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:11:56.832271 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:11:56.832825 (+ 554us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:11:56.833065 (+ 240us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:11:56.833075 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:11:56.834037 (+ 962us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:11:56.834040 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:11:56.834938 (+ 898us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:11:56.834949 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:11:56.835795 (+ 846us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:11:56.835801 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:11:56.836020 (+ 219us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:11:56.836665 (+ 645us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:11:56.836682 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:11:56.840065 (+ 3383us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.235.126:45241: Server oryx/127.6.235.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":361,"thread_start_us":253,"threads_started":1}
W20260928 02:11:56.840440 16348 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:11:56.831390 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.235.67:51881 (local address 127.6.235.126:45241)
0928 02:11:56.831616 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0928 02:11:56.831624 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 02:11:56.831896 (+ 272us) server_negotiation.cc:407] Connection header received
0928 02:11:56.832068 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:11:56.832072 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:11:56.832146 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:11:56.832228 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:11:56.833194 (+ 966us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:11:56.833894 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:11:56.835088 (+ 1194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:11:56.835647 (+ 559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:11:56.835768 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:11:56.840242 (+ 4474us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.235.67:51881: BlockingRecv error: recv got EOF from 127.6.235.67:51881 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
Sep 28 02:11:56 dist-test-slave-86gv krb5kdc[15345](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.235.65@KRBTEST.COM for oryx/127.6.235.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 02:11:56 dist-test-slave-86gv krb5kdc[15345](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.235.65@KRBTEST.COM for oryx/127.6.235.126@KRBTEST.COM, Server not found in Kerberos database
I20260928 02:11:56.992614 16368 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:11:56.984128 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.235.126:45241 (local address 127.6.235.65:59009)
0928 02:11:56.984643 (+ 515us) negotiation.cc:107] Waiting for socket to connect
0928 02:11:56.984666 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0928 02:11:56.984777 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:11:56.985097 (+ 320us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:11:56.985100 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:11:56.985462 (+ 362us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:11:56.985606 (+ 144us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:11:56.985614 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:11:56.986603 (+ 989us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:11:56.986607 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:11:56.987602 (+ 995us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:11:56.987613 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:11:56.988421 (+ 808us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:11:56.988425 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:11:56.988551 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:11:56.989149 (+ 598us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:11:56.989165 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:11:56.992366 (+ 3201us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.235.126:45241: Server oryx/127.6.235.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":397,"thread_start_us":290,"threads_started":1}
W20260928 02:11:56.992839 16348 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:11:56.984218 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.235.65:59009 (local address 127.6.235.126:45241)
0928 02:11:56.984385 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0928 02:11:56.984389 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:11:56.984734 (+ 345us) server_negotiation.cc:407] Connection header received
0928 02:11:56.984869 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:11:56.984873 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:11:56.984950 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:11:56.985033 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:11:56.985724 (+ 691us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:11:56.986462 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:11:56.987758 (+ 1296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:11:56.988253 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:11:56.988385 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:11:56.992604 (+ 4219us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.235.65:59009: BlockingRecv error: recv got EOF from 127.6.235.65:59009 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
Sep 28 02:11:57 dist-test-slave-86gv krb5kdc[15345](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.235.66@KRBTEST.COM for oryx/127.6.235.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 02:11:57 dist-test-slave-86gv krb5kdc[15345](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.235.66@KRBTEST.COM for oryx/127.6.235.126@KRBTEST.COM, Server not found in Kerberos database
I20260928 02:11:57.414180 16369 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:11:57.404652 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.235.126:45241 (local address 127.6.235.66:58055)
0928 02:11:57.405263 (+ 611us) negotiation.cc:107] Waiting for socket to connect
0928 02:11:57.405289 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0928 02:11:57.405440 (+ 151us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:11:57.405803 (+ 363us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:11:57.405806 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:11:57.406209 (+ 403us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:11:57.406367 (+ 158us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:11:57.406377 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:11:57.407391 (+ 1014us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:11:57.407396 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:11:57.408319 (+ 923us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:11:57.408331 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:11:57.409086 (+ 755us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:11:57.409090 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:11:57.409214 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:11:57.409817 (+ 603us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:11:57.409834 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:11:57.413813 (+ 3979us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.235.126:45241: Server oryx/127.6.235.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":505,"thread_start_us":410,"threads_started":1}
W20260928 02:11:57.414228 16348 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:11:57.404734 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.235.66:58055 (local address 127.6.235.126:45241)
0928 02:11:57.404941 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0928 02:11:57.404948 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 02:11:57.405420 (+ 472us) server_negotiation.cc:407] Connection header received
0928 02:11:57.405544 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:11:57.405548 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:11:57.405612 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:11:57.405713 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:11:57.406550 (+ 837us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:11:57.407250 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:11:57.408450 (+ 1200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:11:57.408878 (+ 428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:11:57.408993 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:11:57.414002 (+ 5009us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.235.66:58055: BlockingRecv error: recv got EOF from 127.6.235.66:58055 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
Sep 28 02:11:57 dist-test-slave-86gv krb5kdc[15345](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.235.67@KRBTEST.COM for oryx/127.6.235.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 02:11:57 dist-test-slave-86gv krb5kdc[15345](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.235.67@KRBTEST.COM for oryx/127.6.235.126@KRBTEST.COM, Server not found in Kerberos database
I20260928 02:11:57.851981 16370 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:11:57.841382 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.235.126:45241 (local address 127.6.235.67:43277)
0928 02:11:57.841856 (+ 474us) negotiation.cc:107] Waiting for socket to connect
0928 02:11:57.841874 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0928 02:11:57.842003 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:11:57.842368 (+ 365us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:11:57.842372 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:11:57.842762 (+ 390us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:11:57.842939 (+ 177us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:11:57.842948 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:11:57.844378 (+ 1430us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:11:57.844383 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:11:57.845987 (+ 1604us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:11:57.846005 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:11:57.846809 (+ 804us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:11:57.846816 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:11:57.847029 (+ 213us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:11:57.847711 (+ 682us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:11:57.847729 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:11:57.851616 (+ 3887us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.235.126:45241: Server oryx/127.6.235.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":376,"thread_start_us":265,"threads_started":1}
W20260928 02:11:57.852187 16348 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:11:57.841460 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.235.67:43277 (local address 127.6.235.126:45241)
0928 02:11:57.841697 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0928 02:11:57.841704 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 02:11:57.841967 (+ 263us) server_negotiation.cc:407] Connection header received
0928 02:11:57.842108 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:11:57.842112 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:11:57.842189 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:11:57.842280 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:11:57.843104 (+ 824us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:11:57.844221 (+ 1117us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:11:57.846169 (+ 1948us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:11:57.846624 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:11:57.846888 (+ 264us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:11:57.851893 (+ 5005us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.235.67:43277: BlockingRecv error: recv got EOF from 127.6.235.67:43277 (error 108)
Metrics: {"server-negotiator.queue_time_us":70}
Sep 28 02:11:57 dist-test-slave-86gv krb5kdc[15345](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.235.65@KRBTEST.COM for oryx/127.6.235.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 02:11:58 dist-test-slave-86gv krb5kdc[15345](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.235.65@KRBTEST.COM for oryx/127.6.235.126@KRBTEST.COM, Server not found in Kerberos database
I20260928 02:11:58.003850 16371 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:11:57.993600 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.235.126:45241 (local address 127.6.235.65:49947)
0928 02:11:57.994239 (+ 639us) negotiation.cc:107] Waiting for socket to connect
0928 02:11:57.994256 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0928 02:11:57.994373 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:11:57.994667 (+ 294us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:11:57.994670 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:11:57.995125 (+ 455us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:11:57.995348 (+ 223us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:11:57.995357 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:11:57.996563 (+ 1206us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:11:57.996567 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:11:57.997710 (+ 1143us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:11:57.997723 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:11:57.998545 (+ 822us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:11:57.998549 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:11:57.998663 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:11:57.999254 (+ 591us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:11:57.999271 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:11:58.003468 (+ 4197us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.235.126:45241: Server oryx/127.6.235.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":494,"thread_start_us":284,"threads_started":1}
W20260928 02:11:58.003988 16348 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:11:57.993664 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.235.65:49947 (local address 127.6.235.126:45241)
0928 02:11:57.993846 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0928 02:11:57.993850 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:11:57.994389 (+ 539us) server_negotiation.cc:407] Connection header received
0928 02:11:57.994450 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:11:57.994453 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:11:57.994533 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:11:57.994617 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:11:57.995531 (+ 914us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:11:57.996372 (+ 841us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:11:57.997867 (+ 1495us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:11:57.998375 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:11:57.998487 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:11:58.003764 (+ 5277us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.235.65:49947: BlockingRecv error: recv got EOF from 127.6.235.65:49947 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
Sep 28 02:11:58 dist-test-slave-86gv krb5kdc[15345](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.235.66@KRBTEST.COM for oryx/127.6.235.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 02:11:58 dist-test-slave-86gv krb5kdc[15345](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.235.66@KRBTEST.COM for oryx/127.6.235.126@KRBTEST.COM, Server not found in Kerberos database
I20260928 02:11:58.424927 16372 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:11:58.415295 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.235.126:45241 (local address 127.6.235.66:57085)
0928 02:11:58.415919 (+ 624us) negotiation.cc:107] Waiting for socket to connect
0928 02:11:58.415940 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0928 02:11:58.416051 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:11:58.416380 (+ 329us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:11:58.416385 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:11:58.416773 (+ 388us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:11:58.417073 (+ 300us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:11:58.417085 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:11:58.418112 (+ 1027us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:11:58.418122 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:11:58.419141 (+ 1019us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:11:58.419153 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:11:58.419947 (+ 794us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:11:58.419953 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:11:58.420128 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:11:58.420809 (+ 681us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:11:58.420826 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:11:58.424549 (+ 3723us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.235.126:45241: Server oryx/127.6.235.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":504,"thread_start_us":260,"threads_started":1}
W20260928 02:11:58.425117 16348 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:11:58.415421 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.235.66:57085 (local address 127.6.235.126:45241)
0928 02:11:58.415647 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0928 02:11:58.415651 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:11:58.416018 (+ 367us) server_negotiation.cc:407] Connection header received
0928 02:11:58.416161 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:11:58.416165 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:11:58.416226 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:11:58.416310 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:11:58.417201 (+ 891us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:11:58.417930 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:11:58.419277 (+ 1347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:11:58.419794 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:11:58.419929 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:11:58.424773 (+ 4844us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.235.66:57085: BlockingRecv error: recv got EOF from 127.6.235.66:57085 (error 108)
Metrics: {"server-negotiator.queue_time_us":104}
Sep 28 02:11:58 dist-test-slave-86gv krb5kdc[15345](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.235.67@KRBTEST.COM for oryx/127.6.235.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 02:11:58 dist-test-slave-86gv krb5kdc[15345](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.235.67@KRBTEST.COM for oryx/127.6.235.126@KRBTEST.COM, Server not found in Kerberos database
I20260928 02:11:58.862922 16373 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:11:58.853117 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.235.126:45241 (local address 127.6.235.67:36649)
0928 02:11:58.853761 (+ 644us) negotiation.cc:107] Waiting for socket to connect
0928 02:11:58.853778 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0928 02:11:58.853909 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:11:58.854243 (+ 334us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:11:58.854246 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:11:58.854637 (+ 391us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:11:58.854814 (+ 177us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:11:58.854823 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:11:58.856134 (+ 1311us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:11:58.856139 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:11:58.857213 (+ 1074us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:11:58.857228 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:11:58.857935 (+ 707us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:11:58.857941 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:11:58.858104 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:11:58.858776 (+ 672us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:11:58.858822 (+ 46us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:11:58.862555 (+ 3733us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.235.126:45241: Server oryx/127.6.235.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":534,"thread_start_us":322,"threads_started":1}
W20260928 02:11:58.863006 16348 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:11:58.853220 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.235.67:36649 (local address 127.6.235.126:45241)
0928 02:11:58.853384 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0928 02:11:58.853388 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:11:58.853857 (+ 469us) server_negotiation.cc:407] Connection header received
0928 02:11:58.854042 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:11:58.854046 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:11:58.854106 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:11:58.854201 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:11:58.854965 (+ 764us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:11:58.856003 (+ 1038us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:11:58.857351 (+ 1348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:11:58.857786 (+ 435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:11:58.857916 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:11:58.862774 (+ 4858us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.235.67:36649: BlockingRecv error: recv got EOF from 127.6.235.67:36649 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
Sep 28 02:11:59 dist-test-slave-86gv krb5kdc[15345](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.235.65@KRBTEST.COM for oryx/127.6.235.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 02:11:59 dist-test-slave-86gv krb5kdc[15345](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.235.65@KRBTEST.COM for oryx/127.6.235.126@KRBTEST.COM, Server not found in Kerberos database
I20260928 02:11:59.014518 16374 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:11:59.005017 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.235.126:45241 (local address 127.6.235.65:52455)
0928 02:11:59.005592 (+ 575us) negotiation.cc:107] Waiting for socket to connect
0928 02:11:59.005609 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0928 02:11:59.005729 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:11:59.006052 (+ 323us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:11:59.006056 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:11:59.006474 (+ 418us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:11:59.006669 (+ 195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:11:59.006677 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:11:59.007683 (+ 1006us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:11:59.007688 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:11:59.008745 (+ 1057us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:11:59.008757 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:11:59.009470 (+ 713us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:11:59.009477 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:11:59.009605 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:11:59.010249 (+ 644us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:11:59.010268 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:11:59.014218 (+ 3950us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.235.126:45241: Server oryx/127.6.235.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":412,"thread_start_us":299,"threads_started":1}
W20260928 02:11:59.014866 16348 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:11:59.005176 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.235.65:52455 (local address 127.6.235.126:45241)
0928 02:11:59.005340 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0928 02:11:59.005344 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:11:59.005700 (+ 356us) server_negotiation.cc:407] Connection header received
0928 02:11:59.005826 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:11:59.005829 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:11:59.005897 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:11:59.006001 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:11:59.006797 (+ 796us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:11:59.007552 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:11:59.008880 (+ 1328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:11:59.009306 (+ 426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:11:59.009402 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:11:59.014523 (+ 5121us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.235.65:52455: BlockingRecv error: recv got EOF from 127.6.235.65:52455 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
Sep 28 02:11:59 dist-test-slave-86gv krb5kdc[15345](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.235.66@KRBTEST.COM for oryx/127.6.235.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 02:11:59 dist-test-slave-86gv krb5kdc[15345](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.235.66@KRBTEST.COM for oryx/127.6.235.126@KRBTEST.COM, Server not found in Kerberos database
W20260928 02:11:59.443140 16348 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:11:59.426266 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.235.66:50455 (local address 127.6.235.126:45241)
0928 02:11:59.426571 (+ 305us) server_negotiation.cc:206] Beginning negotiation
0928 02:11:59.426577 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 02:11:59.427016 (+ 439us) server_negotiation.cc:407] Connection header received
0928 02:11:59.427148 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:11:59.427152 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:11:59.427216 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:11:59.427298 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:11:59.428027 (+ 729us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:11:59.429030 (+ 1003us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:11:59.430375 (+ 1345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:11:59.430834 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:11:59.430935 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:11:59.442877 (+ 11942us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.235.66:50455: BlockingRecv error: recv got EOF from 127.6.235.66:50455 (error 108)
Metrics: {"server-negotiator.queue_time_us":85}
I20260928 02:11:59.443128 16375 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:11:59.426204 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.235.126:45241 (local address 127.6.235.66:50455)
0928 02:11:59.426895 (+ 691us) negotiation.cc:107] Waiting for socket to connect
0928 02:11:59.426916 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0928 02:11:59.427056 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:11:59.427359 (+ 303us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:11:59.427362 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:11:59.427744 (+ 382us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:11:59.427902 (+ 158us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:11:59.427910 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:11:59.429186 (+ 1276us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:11:59.429191 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:11:59.430232 (+ 1041us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:11:59.430245 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:11:59.430996 (+ 751us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:11:59.431000 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:11:59.431115 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:11:59.438617 (+ 7502us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:11:59.438649 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:11:59.442612 (+ 3963us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.235.126:45241: Server oryx/127.6.235.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":461,"thread_start_us":290,"threads_started":1}
Sep 28 02:11:59 dist-test-slave-86gv krb5kdc[15345](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.235.67@KRBTEST.COM for oryx/127.6.235.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 02:11:59 dist-test-slave-86gv krb5kdc[15345](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.235.67@KRBTEST.COM for oryx/127.6.235.126@KRBTEST.COM, Server not found in Kerberos database
I20260928 02:11:59.873582 16376 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:11:59.864129 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.235.126:45241 (local address 127.6.235.67:33445)
0928 02:11:59.864703 (+ 574us) negotiation.cc:107] Waiting for socket to connect
0928 02:11:59.864726 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0928 02:11:59.864872 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:11:59.865230 (+ 358us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:11:59.865235 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:11:59.865669 (+ 434us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:11:59.865835 (+ 166us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:11:59.865843 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:11:59.866844 (+ 1001us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:11:59.866850 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:11:59.867791 (+ 941us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:11:59.867802 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:11:59.868577 (+ 775us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:11:59.868584 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:11:59.868747 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:11:59.869415 (+ 668us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:11:59.869433 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:11:59.873317 (+ 3884us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.235.126:45241: Server oryx/127.6.235.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":410,"thread_start_us":285,"threads_started":1}
W20260928 02:11:59.873842 16348 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:11:59.864177 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.235.67:33445 (local address 127.6.235.126:45241)
0928 02:11:59.864428 (+ 251us) server_negotiation.cc:206] Beginning negotiation
0928 02:11:59.864436 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 02:11:59.864799 (+ 363us) server_negotiation.cc:407] Connection header received
0928 02:11:59.865019 (+ 220us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:11:59.865025 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:11:59.865098 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:11:59.865200 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:11:59.865990 (+ 790us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:11:59.866707 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:11:59.867947 (+ 1240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:11:59.868387 (+ 440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:11:59.868526 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:11:59.873558 (+ 5032us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.235.67:33445: BlockingRecv error: recv got EOF from 127.6.235.67:33445 (error 108)
Metrics: {"server-negotiator.queue_time_us":70}
Sep 28 02:12:00 dist-test-slave-86gv krb5kdc[15345](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.235.65@KRBTEST.COM for oryx/127.6.235.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 02:12:00 dist-test-slave-86gv krb5kdc[15345](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.235.65@KRBTEST.COM for oryx/127.6.235.126@KRBTEST.COM, Server not found in Kerberos database
I20260928 02:12:00.025424 16377 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:12:00.015768 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.235.126:45241 (local address 127.6.235.65:50411)
0928 02:12:00.016305 (+ 537us) negotiation.cc:107] Waiting for socket to connect
0928 02:12:00.016322 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0928 02:12:00.016440 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:12:00.016791 (+ 351us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:12:00.016794 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:12:00.017240 (+ 446us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:12:00.017379 (+ 139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:12:00.017388 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:12:00.018473 (+ 1085us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:12:00.018479 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:12:00.019528 (+ 1049us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:12:00.019540 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:12:00.020223 (+ 683us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:12:00.020227 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:12:00.020353 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:12:00.021043 (+ 690us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:12:00.021060 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:12:00.025180 (+ 4120us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.235.126:45241: Server oryx/127.6.235.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":436,"thread_start_us":261,"threads_started":1}
W20260928 02:12:00.025557 16348 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:12:00.015874 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.235.65:50411 (local address 127.6.235.126:45241)
0928 02:12:00.016123 (+ 249us) server_negotiation.cc:206] Beginning negotiation
0928 02:12:00.016131 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 02:12:00.016404 (+ 273us) server_negotiation.cc:407] Connection header received
0928 02:12:00.016597 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:12:00.016601 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:12:00.016665 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:12:00.016755 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:12:00.017554 (+ 799us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:12:00.018297 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:12:00.019685 (+ 1388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:12:00.020088 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:12:00.020187 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:12:00.025360 (+ 5173us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.235.65:50411: BlockingRecv error: recv got EOF from 127.6.235.65:50411 (error 108)
Metrics: {"server-negotiator.queue_time_us":85}
Sep 28 02:12:00 dist-test-slave-86gv krb5kdc[15345](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.235.66@KRBTEST.COM for oryx/127.6.235.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 02:12:00 dist-test-slave-86gv krb5kdc[15345](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.235.66@KRBTEST.COM for oryx/127.6.235.126@KRBTEST.COM, Server not found in Kerberos database
I20260928 02:12:00.453786 16378 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:12:00.444394 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.235.126:45241 (local address 127.6.235.66:50505)
0928 02:12:00.445018 (+ 624us) negotiation.cc:107] Waiting for socket to connect
0928 02:12:00.445033 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 02:12:00.445181 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:12:00.445531 (+ 350us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:12:00.445539 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:12:00.446011 (+ 472us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:12:00.446153 (+ 142us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:12:00.446162 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:12:00.447224 (+ 1062us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:12:00.447230 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:12:00.448151 (+ 921us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:12:00.448162 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:12:00.448876 (+ 714us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:12:00.448882 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:12:00.449081 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:12:00.449686 (+ 605us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:12:00.449704 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:12:00.453533 (+ 3829us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.235.126:45241: Server oryx/127.6.235.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":453,"thread_start_us":251,"threads_started":1}
W20260928 02:12:00.454221 16348 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:12:00.444416 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.235.66:50505 (local address 127.6.235.126:45241)
0928 02:12:00.444634 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0928 02:12:00.444641 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 02:12:00.445111 (+ 470us) server_negotiation.cc:407] Connection header received
0928 02:12:00.445310 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:12:00.445315 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:12:00.445384 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:12:00.445483 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:12:00.446328 (+ 845us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:12:00.447082 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:12:00.448298 (+ 1216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:12:00.448714 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:12:00.448862 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:12:00.453892 (+ 5030us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.235.66:50505: BlockingRecv error: recv got EOF from 127.6.235.66:50505 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
Sep 28 02:12:00 dist-test-slave-86gv krb5kdc[15345](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.235.67@KRBTEST.COM for oryx/127.6.235.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 02:12:00 dist-test-slave-86gv krb5kdc[15345](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.235.67@KRBTEST.COM for oryx/127.6.235.126@KRBTEST.COM, Server not found in Kerberos database
I20260928 02:12:00.883460 16379 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:12:00.874702 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.235.126:45241 (local address 127.6.235.67:47939)
0928 02:12:00.875210 (+ 508us) negotiation.cc:107] Waiting for socket to connect
0928 02:12:00.875225 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 02:12:00.875355 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:12:00.875706 (+ 351us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:12:00.875713 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:12:00.876124 (+ 411us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:12:00.876270 (+ 146us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:12:00.876278 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:12:00.877232 (+ 954us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:12:00.877236 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:12:00.878179 (+ 943us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:12:00.878191 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:12:00.878936 (+ 745us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:12:00.878940 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:12:00.879071 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:12:00.879666 (+ 595us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:12:00.879683 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:12:00.883230 (+ 3547us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.235.126:45241: Server oryx/127.6.235.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":397,"thread_start_us":262,"threads_started":1}
W20260928 02:12:00.883667 16348 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:12:00.874743 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.235.67:47939 (local address 127.6.235.126:45241)
0928 02:12:00.874963 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0928 02:12:00.874970 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 02:12:00.875300 (+ 330us) server_negotiation.cc:407] Connection header received
0928 02:12:00.875467 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:12:00.875470 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:12:00.875557 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:12:00.875658 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:12:00.876388 (+ 730us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:12:00.877124 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:12:00.878300 (+ 1176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:12:00.878713 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:12:00.878808 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:12:00.883424 (+ 4616us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.235.67:47939: BlockingRecv error: recv got EOF from 127.6.235.67:47939 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
Sep 28 02:12:01 dist-test-slave-86gv krb5kdc[15345](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.235.65@KRBTEST.COM for oryx/127.6.235.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 02:12:01 dist-test-slave-86gv krb5kdc[15345](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.235.65@KRBTEST.COM for oryx/127.6.235.126@KRBTEST.COM, Server not found in Kerberos database
I20260928 02:12:01.036052 16380 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:12:01.026695 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.235.126:45241 (local address 127.6.235.65:34491)
0928 02:12:01.027198 (+ 503us) negotiation.cc:107] Waiting for socket to connect
0928 02:12:01.027214 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 02:12:01.027405 (+ 191us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:12:01.027751 (+ 346us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:12:01.027756 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:12:01.028158 (+ 402us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:12:01.028300 (+ 142us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:12:01.028308 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:12:01.029378 (+ 1070us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:12:01.029385 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:12:01.030368 (+ 983us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:12:01.030379 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:12:01.031085 (+ 706us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:12:01.031091 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:12:01.031261 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:12:01.031858 (+ 597us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:12:01.031875 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:12:01.035771 (+ 3896us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.235.126:45241: Server oryx/127.6.235.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":398,"thread_start_us":279,"threads_started":1}
W20260928 02:12:01.036155 16348 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:12:01.026759 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.235.65:34491 (local address 127.6.235.126:45241)
0928 02:12:01.027075 (+ 316us) server_negotiation.cc:206] Beginning negotiation
0928 02:12:01.027081 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 02:12:01.027305 (+ 224us) server_negotiation.cc:407] Connection header received
0928 02:12:01.027534 (+ 229us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:12:01.027543 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:12:01.027619 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:12:01.027722 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:12:01.028455 (+ 733us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:12:01.029241 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:12:01.030509 (+ 1268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:12:01.030934 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:12:01.031058 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:12:01.035941 (+ 4883us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.235.65:34491: BlockingRecv error: recv got EOF from 127.6.235.65:34491 (error 108)
Metrics: {"server-negotiator.queue_time_us":160}
Sep 28 02:12:01 dist-test-slave-86gv krb5kdc[15345](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.235.66@KRBTEST.COM for oryx/127.6.235.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 02:12:01 dist-test-slave-86gv krb5kdc[15345](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.235.66@KRBTEST.COM for oryx/127.6.235.126@KRBTEST.COM, Server not found in Kerberos database
I20260928 02:12:01.463990 16381 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:12:01.455045 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.235.126:45241 (local address 127.6.235.66:57859)
0928 02:12:01.455524 (+ 479us) negotiation.cc:107] Waiting for socket to connect
0928 02:12:01.455540 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 02:12:01.455707 (+ 167us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:12:01.456068 (+ 361us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:12:01.456073 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:12:01.456561 (+ 488us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:12:01.456705 (+ 144us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:12:01.456715 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:12:01.457726 (+ 1011us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:12:01.457734 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:12:01.458709 (+ 975us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:12:01.458720 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:12:01.459402 (+ 682us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:12:01.459407 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:12:01.459576 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:12:01.460159 (+ 583us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:12:01.460177 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:12:01.463618 (+ 3441us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.235.126:45241: Server oryx/127.6.235.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":388,"thread_start_us":279,"threads_started":1}
W20260928 02:12:01.464075 16348 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:12:01.455124 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.235.66:57859 (local address 127.6.235.126:45241)
0928 02:12:01.455327 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0928 02:12:01.455335 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 02:12:01.455637 (+ 302us) server_negotiation.cc:407] Connection header received
0928 02:12:01.455838 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:12:01.455842 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:12:01.455901 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:12:01.456007 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:12:01.456850 (+ 843us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:12:01.457597 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:12:01.458847 (+ 1250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:12:01.459246 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:12:01.459373 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:12:01.463832 (+ 4459us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.235.66:57859: BlockingRecv error: recv got EOF from 127.6.235.66:57859 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
Sep 28 02:12:01 dist-test-slave-86gv krb5kdc[15345](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.235.67@KRBTEST.COM for oryx/127.6.235.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 02:12:01 dist-test-slave-86gv krb5kdc[15345](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.235.67@KRBTEST.COM for oryx/127.6.235.126@KRBTEST.COM, Server not found in Kerberos database
I20260928 02:12:01.893836 16382 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:12:01.884461 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.235.126:45241 (local address 127.6.235.67:38957)
0928 02:12:01.885007 (+ 546us) negotiation.cc:107] Waiting for socket to connect
0928 02:12:01.885023 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 02:12:01.885140 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:12:01.885515 (+ 375us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:12:01.885523 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:12:01.885897 (+ 374us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:12:01.886108 (+ 211us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:12:01.886120 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:12:01.887531 (+ 1411us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:12:01.887534 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:12:01.888494 (+ 960us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:12:01.888506 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:12:01.889289 (+ 783us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:12:01.889293 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:12:01.889401 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:12:01.890001 (+ 600us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:12:01.890018 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:12:01.893505 (+ 3487us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.235.126:45241: Server oryx/127.6.235.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":402,"thread_start_us":290,"threads_started":1}
W20260928 02:12:01.893922 16348 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:12:01.884661 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.235.67:38957 (local address 127.6.235.126:45241)
0928 02:12:01.884833 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0928 02:12:01.884837 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:12:01.885121 (+ 284us) server_negotiation.cc:407] Connection header received
0928 02:12:01.885259 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:12:01.885263 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:12:01.885327 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:12:01.885416 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:12:01.886290 (+ 874us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:12:01.887407 (+ 1117us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:12:01.888636 (+ 1229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:12:01.889164 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:12:01.889264 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:12:01.893679 (+ 4415us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.235.67:38957: BlockingRecv error: recv got EOF from 127.6.235.67:38957 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
Sep 28 02:12:02 dist-test-slave-86gv krb5kdc[15345](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.235.65@KRBTEST.COM for oryx/127.6.235.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 02:12:02 dist-test-slave-86gv krb5kdc[15345](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.235.65@KRBTEST.COM for oryx/127.6.235.126@KRBTEST.COM, Server not found in Kerberos database
I20260928 02:12:02.046574 16383 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:12:02.037062 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.235.126:45241 (local address 127.6.235.65:53013)
0928 02:12:02.037521 (+ 459us) negotiation.cc:107] Waiting for socket to connect
0928 02:12:02.037538 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0928 02:12:02.037672 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:12:02.037969 (+ 297us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:12:02.037972 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:12:02.038352 (+ 380us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:12:02.038551 (+ 199us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:12:02.038559 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:12:02.039575 (+ 1016us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:12:02.039579 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:12:02.040532 (+ 953us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:12:02.040545 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:12:02.041328 (+ 783us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:12:02.041332 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:12:02.041463 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:12:02.042149 (+ 686us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:12:02.042166 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:12:02.046229 (+ 4063us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.235.126:45241: Server oryx/127.6.235.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":331,"thread_start_us":239,"threads_started":1}
W20260928 02:12:02.046706 16348 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:12:02.037115 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.235.65:53013 (local address 127.6.235.126:45241)
0928 02:12:02.037350 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0928 02:12:02.037357 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 02:12:02.037632 (+ 275us) server_negotiation.cc:407] Connection header received
0928 02:12:02.037778 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:12:02.037782 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:12:02.037844 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:12:02.037926 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:12:02.038681 (+ 755us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:12:02.039403 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:12:02.040677 (+ 1274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:12:02.041162 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:12:02.041257 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:12:02.046501 (+ 5244us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.235.65:53013: BlockingRecv error: recv got EOF from 127.6.235.65:53013 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
Sep 28 02:12:02 dist-test-slave-86gv krb5kdc[15345](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.235.66@KRBTEST.COM for oryx/127.6.235.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 02:12:02 dist-test-slave-86gv krb5kdc[15345](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.235.66@KRBTEST.COM for oryx/127.6.235.126@KRBTEST.COM, Server not found in Kerberos database
I20260928 02:12:02.474347 16384 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:12:02.465145 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.235.126:45241 (local address 127.6.235.66:38333)
0928 02:12:02.465691 (+ 546us) negotiation.cc:107] Waiting for socket to connect
0928 02:12:02.465717 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0928 02:12:02.465863 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:12:02.466232 (+ 369us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:12:02.466237 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:12:02.466701 (+ 464us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:12:02.466876 (+ 175us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:12:02.466885 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:12:02.467928 (+ 1043us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:12:02.467934 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:12:02.468855 (+ 921us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:12:02.468867 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:12:02.469576 (+ 709us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:12:02.469582 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:12:02.469752 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:12:02.470382 (+ 630us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:12:02.470399 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:12:02.474020 (+ 3621us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.235.126:45241: Server oryx/127.6.235.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":413,"thread_start_us":299,"threads_started":1}
W20260928 02:12:02.474457 16348 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:12:02.465217 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.235.66:38333 (local address 127.6.235.126:45241)
0928 02:12:02.465452 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0928 02:12:02.465462 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 02:12:02.465823 (+ 361us) server_negotiation.cc:407] Connection header received
0928 02:12:02.466016 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:12:02.466020 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:12:02.466084 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:12:02.466202 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:12:02.467023 (+ 821us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:12:02.467794 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:12:02.469020 (+ 1226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:12:02.469421 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:12:02.469546 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:12:02.474210 (+ 4664us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.235.66:38333: BlockingRecv error: recv got EOF from 127.6.235.66:38333 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
Sep 28 02:12:02 dist-test-slave-86gv krb5kdc[15345](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.235.67@KRBTEST.COM for oryx/127.6.235.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 02:12:02 dist-test-slave-86gv krb5kdc[15345](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.235.67@KRBTEST.COM for oryx/127.6.235.126@KRBTEST.COM, Server not found in Kerberos database
I20260928 02:12:02.903465 16385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:12:02.894730 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.235.126:45241 (local address 127.6.235.67:51697)
0928 02:12:02.895452 (+ 722us) negotiation.cc:107] Waiting for socket to connect
0928 02:12:02.895469 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0928 02:12:02.895589 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:12:02.895931 (+ 342us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:12:02.895936 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:12:02.896383 (+ 447us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:12:02.896596 (+ 213us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:12:02.896605 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:12:02.897567 (+ 962us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:12:02.897570 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:12:02.898465 (+ 895us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:12:02.898476 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:12:02.899121 (+ 645us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:12:02.899125 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:12:02.899240 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:12:02.899825 (+ 585us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:12:02.899842 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:12:02.903200 (+ 3358us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.235.126:45241: Server oryx/127.6.235.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":623,"thread_start_us":295,"threads_started":1}
W20260928 02:12:02.903635 16348 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:12:02.895034 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.235.67:51697 (local address 127.6.235.126:45241)
0928 02:12:02.895239 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0928 02:12:02.895244 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 02:12:02.895541 (+ 297us) server_negotiation.cc:407] Connection header received
0928 02:12:02.895709 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:12:02.895712 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:12:02.895777 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:12:02.895858 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:12:02.896719 (+ 861us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:12:02.897439 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:12:02.898583 (+ 1144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:12:02.899000 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:12:02.899106 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:12:02.903359 (+ 4253us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.235.67:51697: BlockingRecv error: recv got EOF from 127.6.235.67:51697 (error 108)
Metrics: {"server-negotiator.queue_time_us":70}
Sep 28 02:12:03 dist-test-slave-86gv krb5kdc[15345](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.235.65@KRBTEST.COM for oryx/127.6.235.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 02:12:03 dist-test-slave-86gv krb5kdc[15345](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.235.65@KRBTEST.COM for oryx/127.6.235.126@KRBTEST.COM, Server not found in Kerberos database
I20260928 02:12:03.056815 16386 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:12:03.047624 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.235.126:45241 (local address 127.6.235.65:57511)
0928 02:12:03.048139 (+ 515us) negotiation.cc:107] Waiting for socket to connect
0928 02:12:03.048155 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 02:12:03.048271 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:12:03.048548 (+ 277us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:12:03.048551 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:12:03.049046 (+ 495us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:12:03.049210 (+ 164us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:12:03.049219 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:12:03.050237 (+ 1018us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:12:03.050241 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:12:03.051195 (+ 954us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:12:03.051208 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:12:03.051881 (+ 673us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:12:03.051885 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:12:03.052034 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:12:03.052697 (+ 663us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:12:03.052715 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:12:03.056464 (+ 3749us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.235.126:45241: Server oryx/127.6.235.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":396,"thread_start_us":296,"threads_started":1}
W20260928 02:12:03.056913 16348 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:12:03.047706 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.235.65:57511 (local address 127.6.235.126:45241)
0928 02:12:03.047976 (+ 270us) server_negotiation.cc:206] Beginning negotiation
0928 02:12:03.047983 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 02:12:03.048235 (+ 252us) server_negotiation.cc:407] Connection header received
0928 02:12:03.048357 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:12:03.048360 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:12:03.048421 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:12:03.048508 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:12:03.049333 (+ 825us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:12:03.050084 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:12:03.051322 (+ 1238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:12:03.051756 (+ 434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:12:03.051849 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:12:03.056653 (+ 4804us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.235.65:57511: BlockingRecv error: recv got EOF from 127.6.235.65:57511 (error 108)
Metrics: {"server-negotiator.queue_time_us":67}
Sep 28 02:12:03 dist-test-slave-86gv krb5kdc[15345](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.235.66@KRBTEST.COM for oryx/127.6.235.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 02:12:03 dist-test-slave-86gv krb5kdc[15345](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.235.66@KRBTEST.COM for oryx/127.6.235.126@KRBTEST.COM, Server not found in Kerberos database
I20260928 02:12:03.485605 16387 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:12:03.475388 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.235.126:45241 (local address 127.6.235.66:43587)
0928 02:12:03.476401 (+ 1013us) negotiation.cc:107] Waiting for socket to connect
0928 02:12:03.476419 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0928 02:12:03.476598 (+ 179us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:12:03.477285 (+ 687us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:12:03.477290 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:12:03.477714 (+ 424us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:12:03.477872 (+ 158us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:12:03.477881 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:12:03.478963 (+ 1082us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:12:03.478967 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:12:03.480053 (+ 1086us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:12:03.480065 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:12:03.480741 (+ 676us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:12:03.480745 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:12:03.480879 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:12:03.481679 (+ 800us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:12:03.481696 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:12:03.485350 (+ 3654us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.235.126:45241: Server oryx/127.6.235.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":882,"thread_start_us":741,"threads_started":1}
W20260928 02:12:03.485805 16348 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:12:03.475671 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.235.66:43587 (local address 127.6.235.126:45241)
0928 02:12:03.475895 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0928 02:12:03.475902 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 02:12:03.476513 (+ 611us) server_negotiation.cc:407] Connection header received
0928 02:12:03.476758 (+ 245us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:12:03.476763 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:12:03.477117 (+ 354us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:12:03.477229 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:12:03.478041 (+ 812us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:12:03.478798 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:12:03.480183 (+ 1385us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:12:03.480613 (+ 430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:12:03.480717 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:12:03.485574 (+ 4857us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.235.66:43587: BlockingRecv error: recv got EOF from 127.6.235.66:43587 (error 108)
Metrics: {"server-negotiator.queue_time_us":78}
Sep 28 02:12:03 dist-test-slave-86gv krb5kdc[15345](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.235.67@KRBTEST.COM for oryx/127.6.235.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 02:12:03 dist-test-slave-86gv krb5kdc[15345](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.235.67@KRBTEST.COM for oryx/127.6.235.126@KRBTEST.COM, Server not found in Kerberos database
I20260928 02:12:03.913887 16388 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:12:03.904501 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.235.126:45241 (local address 127.6.235.67:33735)
0928 02:12:03.905084 (+ 583us) negotiation.cc:107] Waiting for socket to connect
0928 02:12:03.905100 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 02:12:03.905266 (+ 166us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:12:03.905613 (+ 347us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:12:03.905616 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:12:03.906017 (+ 401us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:12:03.906183 (+ 166us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:12:03.906193 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:12:03.907329 (+ 1136us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:12:03.907333 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:12:03.908312 (+ 979us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:12:03.908326 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:12:03.909192 (+ 866us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:12:03.909196 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:12:03.909326 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:12:03.910038 (+ 712us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:12:03.910056 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:12:03.913645 (+ 3589us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.235.126:45241: Server oryx/127.6.235.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":469,"thread_start_us":309,"threads_started":1}
W20260928 02:12:03.914150 16348 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:12:03.904583 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.235.67:33735 (local address 127.6.235.126:45241)
0928 02:12:03.904780 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0928 02:12:03.904784 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:12:03.905229 (+ 445us) server_negotiation.cc:407] Connection header received
0928 02:12:03.905378 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:12:03.905382 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:12:03.905470 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:12:03.905548 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:12:03.906377 (+ 829us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:12:03.907150 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:12:03.908456 (+ 1306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:12:03.909012 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:12:03.909113 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:12:03.913864 (+ 4751us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.235.67:33735: BlockingRecv error: recv got EOF from 127.6.235.67:33735 (error 108)
Metrics: {"server-negotiator.queue_time_us":79}
Sep 28 02:12:04 dist-test-slave-86gv krb5kdc[15345](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.235.65@KRBTEST.COM for oryx/127.6.235.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 02:12:04 dist-test-slave-86gv krb5kdc[15345](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.235.65@KRBTEST.COM for oryx/127.6.235.126@KRBTEST.COM, Server not found in Kerberos database
I20260928 02:12:04.068142 16389 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:12:04.057947 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.235.126:45241 (local address 127.6.235.65:52143)
0928 02:12:04.058460 (+ 513us) negotiation.cc:107] Waiting for socket to connect
0928 02:12:04.058482 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0928 02:12:04.058666 (+ 184us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:12:04.059191 (+ 525us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:12:04.059194 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:12:04.059723 (+ 529us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:12:04.059864 (+ 141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:12:04.059872 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:12:04.060953 (+ 1081us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:12:04.060958 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:12:04.061859 (+ 901us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:12:04.061870 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:12:04.062601 (+ 731us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:12:04.062607 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:12:04.062780 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:12:04.063420 (+ 640us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:12:04.063441 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:12:04.067696 (+ 4255us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.235.126:45241: Server oryx/127.6.235.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":404,"thread_start_us":299,"threads_started":1}
W20260928 02:12:04.068163 16348 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:12:04.058232 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.235.65:52143 (local address 127.6.235.126:45241)
0928 02:12:04.058446 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0928 02:12:04.058458 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0928 02:12:04.058593 (+ 135us) server_negotiation.cc:407] Connection header received
0928 02:12:04.058928 (+ 335us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:12:04.058933 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:12:04.059007 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:12:04.059147 (+ 140us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:12:04.060008 (+ 861us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:12:04.060779 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:12:04.062011 (+ 1232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:12:04.062449 (+ 438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:12:04.062589 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:12:04.067934 (+ 5345us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.235.65:52143: BlockingRecv error: recv got EOF from 127.6.235.65:52143 (error 108)
Metrics: {"server-negotiator.queue_time_us":73}
Sep 28 02:12:04 dist-test-slave-86gv krb5kdc[15345](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.235.66@KRBTEST.COM for oryx/127.6.235.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 02:12:04 dist-test-slave-86gv krb5kdc[15345](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.235.66@KRBTEST.COM for oryx/127.6.235.126@KRBTEST.COM, Server not found in Kerberos database
I20260928 02:12:04.500242 16390 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:12:04.486592 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.235.126:45241 (local address 127.6.235.66:42115)
0928 02:12:04.487208 (+ 616us) negotiation.cc:107] Waiting for socket to connect
0928 02:12:04.487229 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0928 02:12:04.487404 (+ 175us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:12:04.487751 (+ 347us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:12:04.487756 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:12:04.488190 (+ 434us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:12:04.488329 (+ 139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:12:04.488338 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:12:04.489460 (+ 1122us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:12:04.489468 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:12:04.490393 (+ 925us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:12:04.490404 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:12:04.491176 (+ 772us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:12:04.491180 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:12:04.491440 (+ 260us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:12:04.496521 (+ 5081us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:12:04.496542 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:12:04.499993 (+ 3451us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.235.126:45241: Server oryx/127.6.235.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":504,"thread_start_us":372,"threads_started":1}
W20260928 02:12:04.500500 16348 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:12:04.486938 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.235.66:42115 (local address 127.6.235.126:45241)
0928 02:12:04.487173 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0928 02:12:04.487179 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 02:12:04.487324 (+ 145us) server_negotiation.cc:407] Connection header received
0928 02:12:04.487540 (+ 216us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:12:04.487545 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:12:04.487612 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:12:04.487719 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:12:04.488490 (+ 771us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:12:04.489315 (+ 825us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:12:04.490545 (+ 1230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:12:04.490976 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:12:04.491127 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:12:04.500227 (+ 9100us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.235.66:42115: BlockingRecv error: recv got EOF from 127.6.235.66:42115 (error 108)
Metrics: {"server-negotiator.queue_time_us":83}
Sep 28 02:12:04 dist-test-slave-86gv krb5kdc[15345](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.235.67@KRBTEST.COM for oryx/127.6.235.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 02:12:04 dist-test-slave-86gv krb5kdc[15345](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.235.67@KRBTEST.COM for oryx/127.6.235.126@KRBTEST.COM, Server not found in Kerberos database
I20260928 02:12:04.925273 16391 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:12:04.914798 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.235.126:45241 (local address 127.6.235.67:56807)
0928 02:12:04.915418 (+ 620us) negotiation.cc:107] Waiting for socket to connect
0928 02:12:04.915460 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0928 02:12:04.915636 (+ 176us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:12:04.915994 (+ 358us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:12:04.916000 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:12:04.916515 (+ 515us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:12:04.916690 (+ 175us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:12:04.916700 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:12:04.917886 (+ 1186us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:12:04.917892 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:12:04.918987 (+ 1095us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:12:04.919000 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:12:04.919738 (+ 738us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:12:04.919745 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:12:04.919984 (+ 239us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:12:04.920630 (+ 646us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:12:04.920656 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:12:04.925028 (+ 4372us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.235.126:45241: Server oryx/127.6.235.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":510,"thread_start_us":386,"threads_started":1}
W20260928 02:12:04.925480 16348 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:12:04.915048 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.235.67:56807 (local address 127.6.235.126:45241)
0928 02:12:04.915392 (+ 344us) server_negotiation.cc:206] Beginning negotiation
0928 02:12:04.915398 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 02:12:04.915546 (+ 148us) server_negotiation.cc:407] Connection header received
0928 02:12:04.915778 (+ 232us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:12:04.915789 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:12:04.915853 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:12:04.915960 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:12:04.916855 (+ 895us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:12:04.917737 (+ 882us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:12:04.919139 (+ 1402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:12:04.919571 (+ 432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:12:04.919740 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:12:04.925231 (+ 5491us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.235.67:56807: BlockingRecv error: recv got EOF from 127.6.235.67:56807 (error 108)
Metrics: {"server-negotiator.queue_time_us":200}
Sep 28 02:12:05 dist-test-slave-86gv krb5kdc[15345](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.235.65@KRBTEST.COM for oryx/127.6.235.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 02:12:05 dist-test-slave-86gv krb5kdc[15345](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.235.65@KRBTEST.COM for oryx/127.6.235.126@KRBTEST.COM, Server not found in Kerberos database
I20260928 02:12:05.078702 16392 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:12:05.069616 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.235.126:45241 (local address 127.6.235.65:44025)
0928 02:12:05.070098 (+ 482us) negotiation.cc:107] Waiting for socket to connect
0928 02:12:05.070120 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0928 02:12:05.070274 (+ 154us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:12:05.070595 (+ 321us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:12:05.070600 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:12:05.071009 (+ 409us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:12:05.071209 (+ 200us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:12:05.071221 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:12:05.072315 (+ 1094us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:12:05.072320 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:12:05.073340 (+ 1020us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:12:05.073352 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:12:05.074070 (+ 718us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:12:05.074077 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:12:05.074258 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:12:05.074910 (+ 652us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:12:05.074930 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:12:05.078319 (+ 3389us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.235.126:45241: Server oryx/127.6.235.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":313,"thread_start_us":203,"threads_started":1}
W20260928 02:12:05.078819 16348 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:12:05.069726 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.235.65:44025 (local address 127.6.235.126:45241)
0928 02:12:05.070142 (+ 416us) server_negotiation.cc:206] Beginning negotiation
0928 02:12:05.070151 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 02:12:05.070197 (+ 46us) server_negotiation.cc:407] Connection header received
0928 02:12:05.070399 (+ 202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:12:05.070405 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:12:05.070471 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:12:05.070571 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:12:05.071365 (+ 794us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:12:05.072178 (+ 813us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:12:05.073488 (+ 1310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:12:05.073925 (+ 437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:12:05.074057 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:12:05.078538 (+ 4481us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.235.65:44025: BlockingRecv error: recv got EOF from 127.6.235.65:44025 (error 108)
Metrics: {"server-negotiator.queue_time_us":147}
Sep 28 02:12:05 dist-test-slave-86gv krb5kdc[15345](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.235.66@KRBTEST.COM for oryx/127.6.235.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 02:12:05 dist-test-slave-86gv krb5kdc[15345](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.235.66@KRBTEST.COM for oryx/127.6.235.126@KRBTEST.COM, Server not found in Kerberos database
I20260928 02:12:05.511883 16393 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:12:05.501746 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.235.126:45241 (local address 127.6.235.66:60725)
0928 02:12:05.502314 (+ 568us) negotiation.cc:107] Waiting for socket to connect
0928 02:12:05.502336 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0928 02:12:05.502514 (+ 178us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:12:05.503048 (+ 534us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:12:05.503053 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:12:05.503539 (+ 486us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:12:05.503788 (+ 249us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:12:05.503798 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:12:05.505095 (+ 1297us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:12:05.505100 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:12:05.506184 (+ 1084us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:12:05.506198 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:12:05.506954 (+ 756us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:12:05.506959 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:12:05.507169 (+ 210us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:12:05.507955 (+ 786us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:12:05.507974 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:12:05.511642 (+ 3668us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.235.126:45241: Server oryx/127.6.235.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":410,"thread_start_us":279,"threads_started":1}
W20260928 02:12:05.512097 16348 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:12:05.501886 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.235.66:60725 (local address 127.6.235.126:45241)
0928 02:12:05.502111 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0928 02:12:05.502118 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 02:12:05.502486 (+ 368us) server_negotiation.cc:407] Connection header received
0928 02:12:05.502702 (+ 216us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:12:05.502708 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:12:05.502789 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:12:05.502978 (+ 189us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:12:05.503954 (+ 976us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:12:05.504951 (+ 997us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:12:05.506368 (+ 1417us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:12:05.506802 (+ 434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:12:05.506902 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:12:05.511845 (+ 4943us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.235.66:60725: BlockingRecv error: recv got EOF from 127.6.235.66:60725 (error 108)
Metrics: {"server-negotiator.queue_time_us":69}
I20260928 02:12:05.563097 7085 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskTlaUuc/build/asan/bin/kudu pid 15933
I20260928 02:12:05.576614 16395 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:12:05.564454 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53744 (local address 127.6.235.65:36415)
0928 02:12:05.565201 (+ 747us) server_negotiation.cc:206] Beginning negotiation
0928 02:12:05.565208 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 02:12:05.565239 (+ 31us) server_negotiation.cc:407] Connection header received
0928 02:12:05.565310 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:12:05.565316 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:12:05.565584 (+ 268us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:12:05.565761 (+ 177us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:12:05.567346 (+ 1585us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:12:05.568496 (+ 1150us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:12:05.569876 (+ 1380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:12:05.570570 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:12:05.570671 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:12:05.573488 (+ 2817us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:12:05.573497 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:12:05.573512 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:12:05.573564 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:12:05.576069 (+ 2505us) server_negotiation.cc:394] Sending RPC error: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.6.235.65@KRBTEST.COM not found in keytab (ticket kvno 2)
0928 02:12:05.576377 (+ 308us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:53744: Request ticket server kudu/127.6.235.65@KRBTEST.COM not found in keytab (ticket kvno 2)
Metrics: {"server-negotiator.queue_time_us":559,"thread_start_us":399,"threads_started":1}
W20260928 02:12:05.576788 7085 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskTlaUuc/build/asan/bin/kudu pid 15933: Not authorized: Client connection negotiation failed: client connection to 127.6.235.65:36415: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.6.235.65@KRBTEST.COM not found in keytab (ticket kvno 2)
I20260928 02:12:05.577036 7085 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTlaUuc/build/asan/bin/kudu with pid 15933
I20260928 02:12:05.595129 7085 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskTlaUuc/build/asan/bin/kudu pid 16084
I20260928 02:12:05.607956 16396 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:12:05.596293 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46572 (local address 127.6.235.66:36665)
0928 02:12:05.597093 (+ 800us) server_negotiation.cc:206] Beginning negotiation
0928 02:12:05.597102 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 02:12:05.597138 (+ 36us) server_negotiation.cc:407] Connection header received
0928 02:12:05.597212 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:12:05.597222 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:12:05.597571 (+ 349us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:12:05.597796 (+ 225us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:12:05.599095 (+ 1299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:12:05.600278 (+ 1183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:12:05.601693 (+ 1415us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:12:05.602274 (+ 581us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:12:05.602374 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:12:05.604803 (+ 2429us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:12:05.604813 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:12:05.604828 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:12:05.604869 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:12:05.607366 (+ 2497us) server_negotiation.cc:394] Sending RPC error: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.6.235.66@KRBTEST.COM not found in keytab (ticket kvno 2)
0928 02:12:05.607676 (+ 310us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:46572: Request ticket server kudu/127.6.235.66@KRBTEST.COM not found in keytab (ticket kvno 2)
Metrics: {"server-negotiator.queue_time_us":556,"thread_start_us":397,"threads_started":1}
W20260928 02:12:05.608089 16396 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:46572: Request ticket server kudu/127.6.235.66@KRBTEST.COM not found in keytab (ticket kvno 2)
W20260928 02:12:05.608084 7085 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskTlaUuc/build/asan/bin/kudu pid 16084: Not authorized: Client connection negotiation failed: client connection to 127.6.235.66:36665: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.6.235.66@KRBTEST.COM not found in keytab (ticket kvno 2)
I20260928 02:12:05.608244 7085 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTlaUuc/build/asan/bin/kudu with pid 16084
I20260928 02:12:05.627631 7085 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskTlaUuc/build/asan/bin/kudu pid 16217
I20260928 02:12:05.644290 16397 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:12:05.628848 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52022 (local address 127.6.235.67:38441)
0928 02:12:05.633399 (+ 4551us) server_negotiation.cc:206] Beginning negotiation
0928 02:12:05.633407 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 02:12:05.633438 (+ 31us) server_negotiation.cc:407] Connection header received
0928 02:12:05.633513 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:12:05.633520 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:12:05.633779 (+ 259us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:12:05.634025 (+ 246us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:12:05.635430 (+ 1405us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:12:05.636449 (+ 1019us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:12:05.637733 (+ 1284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:12:05.638305 (+ 572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:12:05.638460 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:12:05.641048 (+ 2588us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:12:05.641057 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:12:05.641071 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:12:05.641113 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:12:05.643522 (+ 2409us) server_negotiation.cc:394] Sending RPC error: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.6.235.67@KRBTEST.COM not found in keytab (ticket kvno 2)
0928 02:12:05.643943 (+ 421us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:52022: Request ticket server kudu/127.6.235.67@KRBTEST.COM not found in keytab (ticket kvno 2)
Metrics: {"server-negotiator.queue_time_us":4368,"thread_start_us":4228,"threads_started":1}
W20260928 02:12:05.644306 7085 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskTlaUuc/build/asan/bin/kudu pid 16217: Not authorized: Client connection negotiation failed: client connection to 127.6.235.67:38441: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.6.235.67@KRBTEST.COM not found in keytab (ticket kvno 2)
W20260928 02:12:05.644407 16397 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:52022: Request ticket server kudu/127.6.235.67@KRBTEST.COM not found in keytab (ticket kvno 2)
I20260928 02:12:05.644461 7085 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTlaUuc/build/asan/bin/kudu with pid 16217
I20260928 02:12:05.661983 7085 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskTlaUuc/build/asan/bin/kudu pid 15864
I20260928 02:12:05.663465 15926 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 02:12:05.777788 7085 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTlaUuc/build/asan/bin/kudu with pid 15864
2026-09-28T02:12:05Z chronyd exiting
[ OK ] SecurityITest.TestMismatchingPrincipals (21552 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationInsecureCluster
2026-09-28T02:12:05Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T02:12:05Z Disabled control of system clock
I20260928 02:12:05.857453 7085 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTlaUuc/build/asan/bin/kudu
/tmp/dist-test-taskTlaUuc/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790561432251112-7085-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790561432251112-7085-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790561432251112-7085-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790561432251112-7085-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.235.126:37097
--webserver_interface=127.6.235.126
--webserver_port=0
--builtin_ntp_servers=127.6.235.84:42925
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.235.126:37097
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260928 02:12:06.202057 16402 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:12:06.202445 16402 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:12:06.202535 16402 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:12:06.210556 16402 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 02:12:06.210678 16402 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:12:06.210716 16402 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:12:06.210777 16402 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 02:12:06.210822 16402 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 02:12:06.220328 16402 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.235.84:42925
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790561432251112-7085-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790561432251112-7085-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.235.126:37097
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.235.126:37097
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790561432251112-7085-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.235.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790561432251112-7085-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 00:51:59 UTC on ccycloud.quasar-axgkwn.root.comops.site
build id 4676464
ASAN enabled
I20260928 02:12:06.221664 16402 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:12:06.223412 16402 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:12:06.236486 16407 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:12:06.236515 16408 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:12:06.236557 16410 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:12:06.236965 16402 server_base.cc:1034] running on GCE node
I20260928 02:12:06.237855 16402 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:12:06.239524 16402 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:12:06.240875 16402 hybrid_clock.cc:648] HybridClock initialized: now 1790561526240821 us; error 52 us; skew 500 ppm
I20260928 02:12:06.241390 16402 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260928 02:12:06.244226 16402 webserver.cc:460] Webserver started at http://127.6.235.126:34141/ using document root <none> and password file <none>
I20260928 02:12:06.245101 16402 fs_manager.cc:359] Metadata directory not provided
I20260928 02:12:06.245242 16402 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:12:06.245600 16402 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:12:06.248667 16402 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790561432251112-7085-0/minicluster-data/master-0/data/instance:
uuid: "b6959b5265204d5985b15d4c4fe70ea4"
format_stamp: "Formatted at 2026-09-28 02:12:06 on dist-test-slave-86gv"
server_key: "7e23746e165c009d042a40315e81b5cd"
server_key_iv: "8349e64572051279553b9eadcbcf23e0"
server_key_version: "encryptionkey@0"
I20260928 02:12:06.249646 16402 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790561432251112-7085-0/minicluster-data/master-0/wal/instance:
uuid: "b6959b5265204d5985b15d4c4fe70ea4"
format_stamp: "Formatted at 2026-09-28 02:12:06 on dist-test-slave-86gv"
server_key: "7e23746e165c009d042a40315e81b5cd"
server_key_iv: "8349e64572051279553b9eadcbcf23e0"
server_key_version: "encryptionkey@0"
I20260928 02:12:06.255467 16402 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.002s
I20260928 02:12:06.259250 16416 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:12:06.260720 16402 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260928 02:12:06.260964 16402 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790561432251112-7085-0/minicluster-data/master-0/data,/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790561432251112-7085-0/minicluster-data/master-0/wal
uuid: "b6959b5265204d5985b15d4c4fe70ea4"
format_stamp: "Formatted at 2026-09-28 02:12:06 on dist-test-slave-86gv"
server_key: "7e23746e165c009d042a40315e81b5cd"
server_key_iv: "8349e64572051279553b9eadcbcf23e0"
server_key_version: "encryptionkey@0"
I20260928 02:12:06.261194 16402 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790561432251112-7085-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790561432251112-7085-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790561432251112-7085-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:12:06.279031 16402 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:12:06.279947 16402 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:12:06.280260 16402 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:12:06.299484 16402 rpc_server.cc:307] RPC server started. Bound to: 127.6.235.126:37097
I20260928 02:12:06.299536 16467 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.235.126:37097 every 8 connection(s)
I20260928 02:12:06.301055 16402 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790561432251112-7085-0/minicluster-data/master-0/data/info.pb
I20260928 02:12:06.305167 16468 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:12:06.311467 7085 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTlaUuc/build/asan/bin/kudu as pid 16402
I20260928 02:12:06.311692 7085 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790561432251112-7085-0/minicluster-data/master-0/wal/instance
I20260928 02:12:06.312373 7085 external_mini_cluster.cc:1428] Setting key 54095e443c762ab72e006a1b74ab9fe7
I20260928 02:12:06.316735 16468 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b6959b5265204d5985b15d4c4fe70ea4: Bootstrap starting.
I20260928 02:12:06.320294 16468 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b6959b5265204d5985b15d4c4fe70ea4: Neither blocks nor log segments found. Creating new log.
I20260928 02:12:06.321651 16468 log.cc:824] T 00000000000000000000000000000000 P b6959b5265204d5985b15d4c4fe70ea4: Log is configured to *not* fsync() on all Append() calls
I20260928 02:12:06.325323 16468 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b6959b5265204d5985b15d4c4fe70ea4: No bootstrap required, opened a new log
I20260928 02:12:06.328048 16471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:12:06.314933 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56406 (local address 127.6.235.126:37097)
0928 02:12:06.315977 (+ 1044us) server_negotiation.cc:206] Beginning negotiation
0928 02:12:06.315990 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0928 02:12:06.316038 (+ 48us) server_negotiation.cc:407] Connection header received
0928 02:12:06.317875 (+ 1837us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:12:06.317902 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:12:06.318445 (+ 543us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:12:06.318803 (+ 358us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:12:06.319497 (+ 694us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:12:06.320960 (+ 1463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:12:06.322828 (+ 1868us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:12:06.323742 (+ 914us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:12:06.323896 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:12:06.325316 (+ 1420us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:12:06.325324 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:12:06.325341 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 02:12:06.326060 (+ 719us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:12:06.326151 (+ 91us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 02:12:06.326175 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:12:06.326278 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0928 02:12:06.326554 (+ 276us) server_negotiation.cc:299] Negotiation successful
0928 02:12:06.326941 (+ 387us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":576,"thread_start_us":364,"threads_started":1}
I20260928 02:12:06.333659 16468 raft_consensus.cc:348] T 00000000000000000000000000000000 P b6959b5265204d5985b15d4c4fe70ea4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b6959b5265204d5985b15d4c4fe70ea4" member_type: VOTER last_known_addr { host: "127.6.235.126" port: 37097 } }
I20260928 02:12:06.334053 16468 raft_consensus.cc:374] T 00000000000000000000000000000000 P b6959b5265204d5985b15d4c4fe70ea4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:12:06.334175 16468 raft_consensus.cc:729] T 00000000000000000000000000000000 P b6959b5265204d5985b15d4c4fe70ea4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b6959b5265204d5985b15d4c4fe70ea4, State: Initialized, Role: FOLLOWER
I20260928 02:12:06.334789 16468 consensus_queue.cc:260] T 00000000000000000000000000000000 P b6959b5265204d5985b15d4c4fe70ea4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b6959b5265204d5985b15d4c4fe70ea4" member_type: VOTER last_known_addr { host: "127.6.235.126" port: 37097 } }
I20260928 02:12:06.335007 16468 raft_consensus.cc:388] T 00000000000000000000000000000000 P b6959b5265204d5985b15d4c4fe70ea4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 02:12:06.335145 16468 raft_consensus.cc:482] T 00000000000000000000000000000000 P b6959b5265204d5985b15d4c4fe70ea4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 02:12:06.335274 16468 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b6959b5265204d5985b15d4c4fe70ea4 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:12:06.337942 16468 raft_consensus.cc:504] T 00000000000000000000000000000000 P b6959b5265204d5985b15d4c4fe70ea4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b6959b5265204d5985b15d4c4fe70ea4" member_type: VOTER last_known_addr { host: "127.6.235.126" port: 37097 } }
I20260928 02:12:06.338457 16468 leader_election.cc:302] T 00000000000000000000000000000000 P b6959b5265204d5985b15d4c4fe70ea4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b6959b5265204d5985b15d4c4fe70ea4; no voters:
I20260928 02:12:06.339111 16468 leader_election.cc:288] T 00000000000000000000000000000000 P b6959b5265204d5985b15d4c4fe70ea4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 02:12:06.339282 16473 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b6959b5265204d5985b15d4c4fe70ea4 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:12:06.340157 16473 raft_consensus.cc:686] T 00000000000000000000000000000000 P b6959b5265204d5985b15d4c4fe70ea4 [term 1 LEADER]: Becoming Leader. State: Replica: b6959b5265204d5985b15d4c4fe70ea4, State: Running, Role: LEADER
I20260928 02:12:06.340876 16473 consensus_queue.cc:237] T 00000000000000000000000000000000 P b6959b5265204d5985b15d4c4fe70ea4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b6959b5265204d5985b15d4c4fe70ea4" member_type: VOTER last_known_addr { host: "127.6.235.126" port: 37097 } }
I20260928 02:12:06.341513 16468 sys_catalog.cc:564] T 00000000000000000000000000000000 P b6959b5265204d5985b15d4c4fe70ea4 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 02:12:06.344230 16474 sys_catalog.cc:455] T 00000000000000000000000000000000 P b6959b5265204d5985b15d4c4fe70ea4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b6959b5265204d5985b15d4c4fe70ea4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b6959b5265204d5985b15d4c4fe70ea4" member_type: VOTER last_known_addr { host: "127.6.235.126" port: 37097 } } }
I20260928 02:12:06.344743 16474 sys_catalog.cc:458] T 00000000000000000000000000000000 P b6959b5265204d5985b15d4c4fe70ea4 [sys.catalog]: This master's current role is: LEADER
I20260928 02:12:06.344259 16475 sys_catalog.cc:455] T 00000000000000000000000000000000 P b6959b5265204d5985b15d4c4fe70ea4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b6959b5265204d5985b15d4c4fe70ea4. Latest consensus state: current_term: 1 leader_uuid: "b6959b5265204d5985b15d4c4fe70ea4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b6959b5265204d5985b15d4c4fe70ea4" member_type: VOTER last_known_addr { host: "127.6.235.126" port: 37097 } } }
I20260928 02:12:06.345220 16475 sys_catalog.cc:458] T 00000000000000000000000000000000 P b6959b5265204d5985b15d4c4fe70ea4 [sys.catalog]: This master's current role is: LEADER
I20260928 02:12:06.346591 16480 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 02:12:06.352357 16480 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 02:12:06.363827 16480 catalog_manager.cc:1347] Generated new cluster ID: 8b93847fd8de4b3496e94fdfb292c975
I20260928 02:12:06.363981 16480 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 02:12:06.387491 16480 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 02:12:06.389345 16480 catalog_manager.cc:1504] Loading token signing keys...
I20260928 02:12:06.400913 16480 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b6959b5265204d5985b15d4c4fe70ea4: Generated new TSK 0
I20260928 02:12:06.401980 16480 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 02:12:06.417239 7085 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTlaUuc/build/asan/bin/kudu
/tmp/dist-test-taskTlaUuc/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790561432251112-7085-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790561432251112-7085-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790561432251112-7085-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790561432251112-7085-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.235.65:0
--local_ip_for_outbound_sockets=127.6.235.65
--webserver_interface=127.6.235.65
--webserver_port=0
--tserver_master_addrs=127.6.235.126:37097
--builtin_ntp_servers=127.6.235.84:42925
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260928 02:12:06.768332 16492 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:12:06.768623 16492 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:12:06.768704 16492 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:12:06.776276 16492 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:12:06.776391 16492 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:12:06.776494 16492 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.235.65
I20260928 02:12:06.786147 16492 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.235.84:42925
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790561432251112-7085-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790561432251112-7085-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.235.65:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790561432251112-7085-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.6.235.65
--webserver_port=0
--tserver_master_addrs=127.6.235.126:37097
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.235.65
--log_dir=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790561432251112-7085-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 00:51:59 UTC on ccycloud.quasar-axgkwn.root.comops.site
build id 4676464
ASAN enabled
I20260928 02:12:06.787542 16492 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:12:06.789351 16492 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:12:06.802934 16497 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:12:06.802990 16500 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:12:06.803006 16498 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:12:06.804039 16492 server_base.cc:1034] running on GCE node
I20260928 02:12:06.804651 16492 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:12:06.805713 16492 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:12:06.806969 16492 hybrid_clock.cc:648] HybridClock initialized: now 1790561526806914 us; error 49 us; skew 500 ppm
I20260928 02:12:06.807294 16492 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260928 02:12:06.809739 16492 webserver.cc:460] Webserver started at http://127.6.235.65:33653/ using document root <none> and password file <none>
I20260928 02:12:06.810489 16492 fs_manager.cc:359] Metadata directory not provided
I20260928 02:12:06.810617 16492 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:12:06.810956 16492 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:12:06.813978 16492 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790561432251112-7085-0/minicluster-data/ts-0/data/instance:
uuid: "aca417bd7a064a64bdcdd396b0a886d1"
format_stamp: "Formatted at 2026-09-28 02:12:06 on dist-test-slave-86gv"
server_key: "a4c0043be523d95389d8299eb6e402b1"
server_key_iv: "fbedb3b003f98460f51b7313bd6342bc"
server_key_version: "encryptionkey@0"
I20260928 02:12:06.814793 16492 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790561432251112-7085-0/minicluster-data/ts-0/wal/instance:
uuid: "aca417bd7a064a64bdcdd396b0a886d1"
format_stamp: "Formatted at 2026-09-28 02:12:06 on dist-test-slave-86gv"
server_key: "a4c0043be523d95389d8299eb6e402b1"
server_key_iv: "fbedb3b003f98460f51b7313bd6342bc"
server_key_version: "encryptionkey@0"
I20260928 02:12:06.820443 16492 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260928 02:12:06.823952 16506 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:12:06.825364 16492 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 02:12:06.825538 16492 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790561432251112-7085-0/minicluster-data/ts-0/data,/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790561432251112-7085-0/minicluster-data/ts-0/wal
uuid: "aca417bd7a064a64bdcdd396b0a886d1"
format_stamp: "Formatted at 2026-09-28 02:12:06 on dist-test-slave-86gv"
server_key: "a4c0043be523d95389d8299eb6e402b1"
server_key_iv: "fbedb3b003f98460f51b7313bd6342bc"
server_key_version: "encryptionkey@0"
I20260928 02:12:06.825764 16492 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790561432251112-7085-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790561432251112-7085-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790561432251112-7085-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:12:06.842419 16492 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:12:06.843321 16492 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:12:06.843573 16492 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:12:06.844458 16492 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:12:06.845965 16492 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:12:06.846087 16492 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:12:06.846215 16492 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:12:06.846297 16492 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:12:06.879050 16492 rpc_server.cc:307] RPC server started. Bound to: 127.6.235.65:37349
I20260928 02:12:06.879087 16618 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.235.65:37349 every 8 connection(s)
I20260928 02:12:06.880369 16492 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790561432251112-7085-0/minicluster-data/ts-0/data/info.pb
I20260928 02:12:06.881816 7085 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTlaUuc/build/asan/bin/kudu as pid 16492
I20260928 02:12:06.882048 7085 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790561432251112-7085-0/minicluster-data/ts-0/wal/instance
I20260928 02:12:06.882707 7085 external_mini_cluster.cc:1428] Setting key 8eea2e11cf09f379a3f203b49cce289b
I20260928 02:12:06.885535 7085 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTlaUuc/build/asan/bin/kudu
/tmp/dist-test-taskTlaUuc/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790561432251112-7085-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790561432251112-7085-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790561432251112-7085-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790561432251112-7085-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.235.66:0
--local_ip_for_outbound_sockets=127.6.235.66
--webserver_interface=127.6.235.66
--webserver_port=0
--tserver_master_addrs=127.6.235.126:37097
--builtin_ntp_servers=127.6.235.84:42925
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260928 02:12:06.894256 16621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:12:06.882929 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.235.65:44497 (local address 127.6.235.126:37097)
0928 02:12:06.883501 (+ 572us) server_negotiation.cc:206] Beginning negotiation
0928 02:12:06.883509 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 02:12:06.884873 (+ 1364us) server_negotiation.cc:407] Connection header received
0928 02:12:06.886744 (+ 1871us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:12:06.886753 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:12:06.886843 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:12:06.886964 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:12:06.888226 (+ 1262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:12:06.889502 (+ 1276us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:12:06.891082 (+ 1580us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:12:06.891781 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:12:06.891971 (+ 190us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:12:06.893364 (+ 1393us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:12:06.893373 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:12:06.893380 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 02:12:06.893406 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:12:06.893465 (+ 59us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 02:12:06.893488 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:12:06.893593 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0928 02:12:06.893816 (+ 223us) server_negotiation.cc:299] Negotiation successful
0928 02:12:06.893925 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":360,"thread_start_us":242,"threads_started":1}
I20260928 02:12:06.895351 16622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:12:06.883633 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.235.126:37097 (local address 127.6.235.65:44497)
0928 02:12:06.884698 (+ 1065us) negotiation.cc:107] Waiting for socket to connect
0928 02:12:06.884750 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0928 02:12:06.886470 (+ 1720us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:12:06.887349 (+ 879us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:12:06.887358 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:12:06.887469 (+ 111us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:12:06.888050 (+ 581us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:12:06.888067 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:12:06.889646 (+ 1579us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:12:06.889650 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:12:06.890922 (+ 1272us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:12:06.890934 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:12:06.892041 (+ 1107us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:12:06.892045 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:12:06.892243 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:12:06.893050 (+ 807us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 02:12:06.893053 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:12:06.893093 (+ 40us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 02:12:06.893174 (+ 81us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:12:06.893593 (+ 419us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:12:06.893601 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:12:06.893607 (+ 6us) client_negotiation.cc:769] Sending connection context
0928 02:12:06.893804 (+ 197us) client_negotiation.cc:240] Negotiation successful
0928 02:12:06.894216 (+ 412us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":504,"thread_start_us":329,"threads_started":1}
I20260928 02:12:06.897099 16619 heartbeater.cc:347] Connected to a master server at 127.6.235.126:37097
I20260928 02:12:06.897408 16619 heartbeater.cc:464] Registering TS with master...
I20260928 02:12:06.898187 16619 heartbeater.cc:511] Master 127.6.235.126:37097 requested a full tablet report, sending...
I20260928 02:12:06.900606 16433 ts_manager.cc:194] Registered new tserver with Master: aca417bd7a064a64bdcdd396b0a886d1 (127.6.235.65:37349)
I20260928 02:12:06.902882 16433 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.6.235.65:44497
W20260928 02:12:07.224596 16623 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:12:07.224941 16623 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:12:07.225046 16623 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:12:07.232565 16623 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:12:07.232671 16623 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:12:07.232775 16623 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.235.66
I20260928 02:12:07.242177 16623 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.235.84:42925
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790561432251112-7085-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790561432251112-7085-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.235.66:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790561432251112-7085-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.6.235.66
--webserver_port=0
--tserver_master_addrs=127.6.235.126:37097
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.235.66
--log_dir=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790561432251112-7085-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 00:51:59 UTC on ccycloud.quasar-axgkwn.root.comops.site
build id 4676464
ASAN enabled
I20260928 02:12:07.243588 16623 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:12:07.245436 16623 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:12:07.259095 16628 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:12:07.259040 16631 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:12:07.259075 16629 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:12:07.260267 16623 server_base.cc:1034] running on GCE node
I20260928 02:12:07.260958 16623 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:12:07.262060 16623 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:12:07.263298 16623 hybrid_clock.cc:648] HybridClock initialized: now 1790561527263242 us; error 43 us; skew 500 ppm
I20260928 02:12:07.263816 16623 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260928 02:12:07.266844 16623 webserver.cc:460] Webserver started at http://127.6.235.66:33025/ using document root <none> and password file <none>
I20260928 02:12:07.267638 16623 fs_manager.cc:359] Metadata directory not provided
I20260928 02:12:07.267732 16623 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:12:07.268057 16623 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:12:07.271420 16623 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790561432251112-7085-0/minicluster-data/ts-1/data/instance:
uuid: "340a6e77f42a440da052e7c895ef8f87"
format_stamp: "Formatted at 2026-09-28 02:12:07 on dist-test-slave-86gv"
server_key: "cc8571c2cddc8739c19fb72372a61db9"
server_key_iv: "126f3cac8fdb320e3ddfed2a2e6fac6a"
server_key_version: "encryptionkey@0"
I20260928 02:12:07.272440 16623 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790561432251112-7085-0/minicluster-data/ts-1/wal/instance:
uuid: "340a6e77f42a440da052e7c895ef8f87"
format_stamp: "Formatted at 2026-09-28 02:12:07 on dist-test-slave-86gv"
server_key: "cc8571c2cddc8739c19fb72372a61db9"
server_key_iv: "126f3cac8fdb320e3ddfed2a2e6fac6a"
server_key_version: "encryptionkey@0"
I20260928 02:12:07.278199 16623 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260928 02:12:07.281975 16637 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:12:07.283465 16623 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260928 02:12:07.283639 16623 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790561432251112-7085-0/minicluster-data/ts-1/data,/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790561432251112-7085-0/minicluster-data/ts-1/wal
uuid: "340a6e77f42a440da052e7c895ef8f87"
format_stamp: "Formatted at 2026-09-28 02:12:07 on dist-test-slave-86gv"
server_key: "cc8571c2cddc8739c19fb72372a61db9"
server_key_iv: "126f3cac8fdb320e3ddfed2a2e6fac6a"
server_key_version: "encryptionkey@0"
I20260928 02:12:07.283874 16623 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790561432251112-7085-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790561432251112-7085-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790561432251112-7085-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:12:07.298751 16623 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:12:07.299635 16623 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:12:07.299950 16623 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:12:07.300964 16623 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:12:07.302522 16623 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:12:07.302608 16623 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:12:07.302770 16623 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:12:07.302858 16623 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:12:07.338449 16623 rpc_server.cc:307] RPC server started. Bound to: 127.6.235.66:37357
I20260928 02:12:07.338505 16749 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.235.66:37357 every 8 connection(s)
I20260928 02:12:07.339826 16623 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790561432251112-7085-0/minicluster-data/ts-1/data/info.pb
I20260928 02:12:07.343098 7085 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTlaUuc/build/asan/bin/kudu as pid 16623
I20260928 02:12:07.343276 7085 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790561432251112-7085-0/minicluster-data/ts-1/wal/instance
I20260928 02:12:07.343865 7085 external_mini_cluster.cc:1428] Setting key e6af5be8e7f6ad13ebb59d09588c3793
I20260928 02:12:07.347231 7085 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTlaUuc/build/asan/bin/kudu
/tmp/dist-test-taskTlaUuc/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790561432251112-7085-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790561432251112-7085-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790561432251112-7085-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790561432251112-7085-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.235.67:0
--local_ip_for_outbound_sockets=127.6.235.67
--webserver_interface=127.6.235.67
--webserver_port=0
--tserver_master_addrs=127.6.235.126:37097
--builtin_ntp_servers=127.6.235.84:42925
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260928 02:12:07.354414 16621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:12:07.342415 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.235.66:53047 (local address 127.6.235.126:37097)
0928 02:12:07.342592 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0928 02:12:07.342597 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 02:12:07.344373 (+ 1776us) server_negotiation.cc:407] Connection header received
0928 02:12:07.346587 (+ 2214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:12:07.346591 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:12:07.346654 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:12:07.346735 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:12:07.348228 (+ 1493us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:12:07.348999 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:12:07.351444 (+ 2445us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:12:07.351832 (+ 388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:12:07.351928 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:12:07.353583 (+ 1655us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:12:07.353588 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:12:07.353591 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 02:12:07.353610 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:12:07.353640 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 02:12:07.353651 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:12:07.353716 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0928 02:12:07.354026 (+ 310us) server_negotiation.cc:299] Negotiation successful
0928 02:12:07.354122 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260928 02:12:07.355767 16752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:12:07.343047 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.235.126:37097 (local address 127.6.235.66:53047)
0928 02:12:07.344195 (+ 1148us) negotiation.cc:107] Waiting for socket to connect
0928 02:12:07.344251 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0928 02:12:07.346287 (+ 2036us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:12:07.347048 (+ 761us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:12:07.347061 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:12:07.347189 (+ 128us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:12:07.348053 (+ 864us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:12:07.348083 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:12:07.349149 (+ 1066us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:12:07.349158 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:12:07.351305 (+ 2147us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:12:07.351322 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:12:07.352058 (+ 736us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:12:07.352067 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:12:07.352388 (+ 321us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:12:07.353223 (+ 835us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 02:12:07.353229 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:12:07.353286 (+ 57us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 02:12:07.353391 (+ 105us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:12:07.353738 (+ 347us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:12:07.353751 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:12:07.353762 (+ 11us) client_negotiation.cc:769] Sending connection context
0928 02:12:07.353964 (+ 202us) client_negotiation.cc:240] Negotiation successful
0928 02:12:07.354400 (+ 436us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":556,"thread_start_us":327,"threads_started":1}
I20260928 02:12:07.357333 16750 heartbeater.cc:347] Connected to a master server at 127.6.235.126:37097
I20260928 02:12:07.357684 16750 heartbeater.cc:464] Registering TS with master...
I20260928 02:12:07.358491 16750 heartbeater.cc:511] Master 127.6.235.126:37097 requested a full tablet report, sending...
I20260928 02:12:07.359951 16433 ts_manager.cc:194] Registered new tserver with Master: 340a6e77f42a440da052e7c895ef8f87 (127.6.235.66:37357)
I20260928 02:12:07.360761 16433 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.6.235.66:53047
W20260928 02:12:07.687813 16753 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:12:07.688167 16753 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:12:07.688323 16753 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:12:07.695741 16753 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:12:07.695914 16753 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:12:07.696102 16753 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.235.67
I20260928 02:12:07.706007 16753 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.235.84:42925
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790561432251112-7085-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790561432251112-7085-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.235.67:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790561432251112-7085-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.6.235.67
--webserver_port=0
--tserver_master_addrs=127.6.235.126:37097
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.235.67
--log_dir=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790561432251112-7085-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 00:51:59 UTC on ccycloud.quasar-axgkwn.root.comops.site
build id 4676464
ASAN enabled
I20260928 02:12:07.707430 16753 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:12:07.709290 16753 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:12:07.723229 16761 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:12:07.723284 16759 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:12:07.723186 16758 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:12:07.724146 16753 server_base.cc:1034] running on GCE node
I20260928 02:12:07.724772 16753 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:12:07.725957 16753 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:12:07.727190 16753 hybrid_clock.cc:648] HybridClock initialized: now 1790561527727117 us; error 71 us; skew 500 ppm
I20260928 02:12:07.727522 16753 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260928 02:12:07.730139 16753 webserver.cc:460] Webserver started at http://127.6.235.67:32939/ using document root <none> and password file <none>
I20260928 02:12:07.730911 16753 fs_manager.cc:359] Metadata directory not provided
I20260928 02:12:07.731003 16753 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:12:07.731273 16753 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:12:07.734403 16753 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790561432251112-7085-0/minicluster-data/ts-2/data/instance:
uuid: "564d46c017e349458540c34e64134f4a"
format_stamp: "Formatted at 2026-09-28 02:12:07 on dist-test-slave-86gv"
server_key: "59ae25f2342f8b7be50c738e808a5a9f"
server_key_iv: "eea238e58ac8fe7f77db28d1e0ad9fc0"
server_key_version: "encryptionkey@0"
I20260928 02:12:07.735190 16753 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790561432251112-7085-0/minicluster-data/ts-2/wal/instance:
uuid: "564d46c017e349458540c34e64134f4a"
format_stamp: "Formatted at 2026-09-28 02:12:07 on dist-test-slave-86gv"
server_key: "59ae25f2342f8b7be50c738e808a5a9f"
server_key_iv: "eea238e58ac8fe7f77db28d1e0ad9fc0"
server_key_version: "encryptionkey@0"
I20260928 02:12:07.740769 16753 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20260928 02:12:07.744772 16767 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:12:07.746265 16753 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 02:12:07.746446 16753 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790561432251112-7085-0/minicluster-data/ts-2/data,/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790561432251112-7085-0/minicluster-data/ts-2/wal
uuid: "564d46c017e349458540c34e64134f4a"
format_stamp: "Formatted at 2026-09-28 02:12:07 on dist-test-slave-86gv"
server_key: "59ae25f2342f8b7be50c738e808a5a9f"
server_key_iv: "eea238e58ac8fe7f77db28d1e0ad9fc0"
server_key_version: "encryptionkey@0"
I20260928 02:12:07.746650 16753 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790561432251112-7085-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790561432251112-7085-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790561432251112-7085-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:12:07.796123 16753 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:12:07.797091 16753 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:12:07.797394 16753 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:12:07.798434 16753 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:12:07.800082 16753 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:12:07.800172 16753 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:12:07.800321 16753 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:12:07.800405 16753 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:12:07.836493 16753 rpc_server.cc:307] RPC server started. Bound to: 127.6.235.67:38045
I20260928 02:12:07.836583 16879 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.235.67:38045 every 8 connection(s)
I20260928 02:12:07.837903 16753 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790561432251112-7085-0/minicluster-data/ts-2/data/info.pb
I20260928 02:12:07.844971 7085 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTlaUuc/build/asan/bin/kudu as pid 16753
I20260928 02:12:07.845176 7085 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790561432251112-7085-0/minicluster-data/ts-2/wal/instance
I20260928 02:12:07.845642 7085 external_mini_cluster.cc:1428] Setting key 73840fd81e05a151cf2659a4aaa070b5
I20260928 02:12:07.852471 16621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:12:07.840612 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.235.67:45577 (local address 127.6.235.126:37097)
0928 02:12:07.840801 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0928 02:12:07.840807 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 02:12:07.842582 (+ 1775us) server_negotiation.cc:407] Connection header received
0928 02:12:07.844406 (+ 1824us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:12:07.844414 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:12:07.844511 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:12:07.844636 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:12:07.845964 (+ 1328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:12:07.846948 (+ 984us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:12:07.849192 (+ 2244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:12:07.849818 (+ 626us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:12:07.849939 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:12:07.851473 (+ 1534us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:12:07.851478 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:12:07.851481 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 02:12:07.851500 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:12:07.851535 (+ 35us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 02:12:07.851548 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:12:07.851625 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0928 02:12:07.852112 (+ 487us) server_negotiation.cc:299] Negotiation successful
0928 02:12:07.852194 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260928 02:12:07.853557 16882 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:12:07.841335 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.235.126:37097 (local address 127.6.235.67:45577)
0928 02:12:07.842425 (+ 1090us) negotiation.cc:107] Waiting for socket to connect
0928 02:12:07.842468 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0928 02:12:07.844109 (+ 1641us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:12:07.844840 (+ 731us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:12:07.844849 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:12:07.845005 (+ 156us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:12:07.845768 (+ 763us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:12:07.845791 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:12:07.847118 (+ 1327us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:12:07.847123 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:12:07.849042 (+ 1919us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:12:07.849054 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:12:07.850026 (+ 972us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:12:07.850031 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:12:07.850326 (+ 295us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:12:07.851092 (+ 766us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 02:12:07.851094 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:12:07.851135 (+ 41us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 02:12:07.851271 (+ 136us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:12:07.851822 (+ 551us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:12:07.851833 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:12:07.851842 (+ 9us) client_negotiation.cc:769] Sending connection context
0928 02:12:07.852048 (+ 206us) client_negotiation.cc:240] Negotiation successful
0928 02:12:07.852398 (+ 350us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":538,"thread_start_us":300,"threads_started":1}
I20260928 02:12:07.854990 16880 heartbeater.cc:347] Connected to a master server at 127.6.235.126:37097
I20260928 02:12:07.855329 16880 heartbeater.cc:464] Registering TS with master...
I20260928 02:12:07.856166 16880 heartbeater.cc:511] Master 127.6.235.126:37097 requested a full tablet report, sending...
I20260928 02:12:07.857708 16433 ts_manager.cc:194] Registered new tserver with Master: 564d46c017e349458540c34e64134f4a (127.6.235.67:38045)
I20260928 02:12:07.858493 16433 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.6.235.67:45577
I20260928 02:12:07.861491 7085 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260928 02:12:07.867728 16621 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:12:07.866102 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56410 (local address 127.6.235.126:37097)
0928 02:12:07.866351 (+ 249us) server_negotiation.cc:206] Beginning negotiation
0928 02:12:07.866358 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 02:12:07.866762 (+ 404us) server_negotiation.cc:407] Connection header received
0928 02:12:07.866933 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:12:07.866937 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:12:07.867012 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:12:07.867105 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:12:07.867493 (+ 388us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56410: BlockingRecv error: recv got EOF from 127.0.0.1:56410 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
I20260928 02:12:07.868561 7085 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskTlaUuc/build/asan/bin/kudu pid 16492
I20260928 02:12:07.876151 16889 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:12:07.869484 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51470 (local address 127.6.235.65:37349)
0928 02:12:07.869967 (+ 483us) server_negotiation.cc:206] Beginning negotiation
0928 02:12:07.869977 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 02:12:07.870007 (+ 30us) server_negotiation.cc:407] Connection header received
0928 02:12:07.870075 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:12:07.870082 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:12:07.870296 (+ 214us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:12:07.870446 (+ 150us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:12:07.870775 (+ 329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:12:07.871775 (+ 1000us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:12:07.873126 (+ 1351us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:12:07.873713 (+ 587us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:12:07.873812 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:12:07.874832 (+ 1020us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:12:07.874841 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:12:07.874858 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 02:12:07.875467 (+ 609us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:12:07.875513 (+ 46us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 02:12:07.875532 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:12:07.875626 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0928 02:12:07.875751 (+ 125us) server_negotiation.cc:299] Negotiation successful
0928 02:12:07.875861 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":338,"thread_start_us":237,"threads_started":1}
I20260928 02:12:07.877475 16614 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 02:12:07.958921 16619 heartbeater.cc:503] Master 127.6.235.126:37097 was elected leader, sending a full tablet report...
I20260928 02:12:07.960088 7085 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTlaUuc/build/asan/bin/kudu with pid 16492
W20260928 02:12:07.977190 16418 connection.cc:535] server connection from 127.6.235.65:44497 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260928 02:12:07.977516 7085 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskTlaUuc/build/asan/bin/kudu pid 16623
I20260928 02:12:07.985666 16891 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:12:07.978431 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40024 (local address 127.6.235.66:37357)
0928 02:12:07.978990 (+ 559us) server_negotiation.cc:206] Beginning negotiation
0928 02:12:07.978997 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 02:12:07.979024 (+ 27us) server_negotiation.cc:407] Connection header received
0928 02:12:07.979091 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:12:07.979097 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:12:07.979318 (+ 221us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:12:07.979457 (+ 139us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:12:07.979854 (+ 397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:12:07.980982 (+ 1128us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:12:07.982235 (+ 1253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:12:07.982864 (+ 629us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:12:07.982966 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:12:07.984066 (+ 1100us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:12:07.984074 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:12:07.984088 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 02:12:07.984787 (+ 699us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:12:07.984837 (+ 50us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 02:12:07.984864 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:12:07.985057 (+ 193us) server_negotiation.cc:1035] Waiting for connection context
0928 02:12:07.985213 (+ 156us) server_negotiation.cc:299] Negotiation successful
0928 02:12:07.985369 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":410,"thread_start_us":287,"threads_started":1}
I20260928 02:12:07.987066 16745 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 02:12:08.070546 7085 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTlaUuc/build/asan/bin/kudu with pid 16623
I20260928 02:12:08.087467 7085 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskTlaUuc/build/asan/bin/kudu pid 16753
I20260928 02:12:08.095752 16893 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:12:08.088560 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46744 (local address 127.6.235.67:38045)
0928 02:12:08.089269 (+ 709us) server_negotiation.cc:206] Beginning negotiation
0928 02:12:08.089276 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 02:12:08.089306 (+ 30us) server_negotiation.cc:407] Connection header received
0928 02:12:08.089367 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:12:08.089374 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:12:08.089774 (+ 400us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:12:08.089933 (+ 159us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:12:08.090321 (+ 388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:12:08.091363 (+ 1042us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:12:08.092506 (+ 1143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:12:08.093166 (+ 660us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:12:08.093265 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:12:08.094304 (+ 1039us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:12:08.094311 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:12:08.094325 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 02:12:08.094938 (+ 613us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:12:08.094982 (+ 44us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 02:12:08.095001 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:12:08.095088 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0928 02:12:08.095216 (+ 128us) server_negotiation.cc:299] Negotiation successful
0928 02:12:08.095363 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":562,"thread_start_us":406,"threads_started":1}
I20260928 02:12:08.097234 16875 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 02:12:08.179674 7085 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTlaUuc/build/asan/bin/kudu with pid 16753
I20260928 02:12:08.196611 7085 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskTlaUuc/build/asan/bin/kudu pid 16402
I20260928 02:12:08.198041 16463 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 02:12:08.301616 7085 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTlaUuc/build/asan/bin/kudu with pid 16402
2026-09-28T02:12:08Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationInsecureCluster (2519 ms)
[ RUN ] SecurityITest.TestRequireEncryptionInsecureCluster
2026-09-28T02:12:08Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T02:12:08Z Disabled control of system clock
I20260928 02:12:08.375685 7085 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTlaUuc/build/asan/bin/kudu
/tmp/dist-test-taskTlaUuc/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790561432251112-7085-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790561432251112-7085-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790561432251112-7085-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790561432251112-7085-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.235.126:41321
--webserver_interface=127.6.235.126
--webserver_port=0
--builtin_ntp_servers=127.6.235.84:38797
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.235.126:41321
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260928 02:12:08.712019 16899 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:12:08.712340 16899 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:12:08.712519 16899 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:12:08.720057 16899 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 02:12:08.720177 16899 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:12:08.720254 16899 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:12:08.720341 16899 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 02:12:08.720418 16899 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 02:12:08.730108 16899 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.235.84:38797
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790561432251112-7085-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790561432251112-7085-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.235.126:41321
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.235.126:41321
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790561432251112-7085-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.235.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790561432251112-7085-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 00:51:59 UTC on ccycloud.quasar-axgkwn.root.comops.site
build id 4676464
ASAN enabled
I20260928 02:12:08.731511 16899 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:12:08.733525 16899 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:12:08.746322 16904 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:12:08.746342 16905 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:12:08.746393 16907 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:12:08.747529 16899 server_base.cc:1034] running on GCE node
I20260928 02:12:08.748405 16899 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:12:08.749970 16899 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:12:08.751209 16899 hybrid_clock.cc:648] HybridClock initialized: now 1790561528751126 us; error 77 us; skew 500 ppm
I20260928 02:12:08.751585 16899 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260928 02:12:08.754006 16899 webserver.cc:460] Webserver started at http://127.6.235.126:36159/ using document root <none> and password file <none>
I20260928 02:12:08.754774 16899 fs_manager.cc:359] Metadata directory not provided
I20260928 02:12:08.754915 16899 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:12:08.755239 16899 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:12:08.758246 16899 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790561432251112-7085-0/minicluster-data/master-0/data/instance:
uuid: "3e9d6610e8d84d4fb0e56aea67127179"
format_stamp: "Formatted at 2026-09-28 02:12:08 on dist-test-slave-86gv"
server_key: "c6912b9c687d7f8d81cb3c3185c589ce"
server_key_iv: "7de2ef449083ceae90cccf2c1f4c7f46"
server_key_version: "encryptionkey@0"
I20260928 02:12:08.759068 16899 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790561432251112-7085-0/minicluster-data/master-0/wal/instance:
uuid: "3e9d6610e8d84d4fb0e56aea67127179"
format_stamp: "Formatted at 2026-09-28 02:12:08 on dist-test-slave-86gv"
server_key: "c6912b9c687d7f8d81cb3c3185c589ce"
server_key_iv: "7de2ef449083ceae90cccf2c1f4c7f46"
server_key_version: "encryptionkey@0"
I20260928 02:12:08.764243 16899 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.003s
I20260928 02:12:08.767879 16913 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:12:08.769327 16899 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 02:12:08.769510 16899 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790561432251112-7085-0/minicluster-data/master-0/data,/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790561432251112-7085-0/minicluster-data/master-0/wal
uuid: "3e9d6610e8d84d4fb0e56aea67127179"
format_stamp: "Formatted at 2026-09-28 02:12:08 on dist-test-slave-86gv"
server_key: "c6912b9c687d7f8d81cb3c3185c589ce"
server_key_iv: "7de2ef449083ceae90cccf2c1f4c7f46"
server_key_version: "encryptionkey@0"
I20260928 02:12:08.769745 16899 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790561432251112-7085-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790561432251112-7085-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790561432251112-7085-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:12:08.775684 16899 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:12:08.776487 16899 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:12:08.776786 16899 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:12:08.795665 16899 rpc_server.cc:307] RPC server started. Bound to: 127.6.235.126:41321
I20260928 02:12:08.795703 16964 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.235.126:41321 every 8 connection(s)
I20260928 02:12:08.797081 16899 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790561432251112-7085-0/minicluster-data/master-0/data/info.pb
I20260928 02:12:08.798473 7085 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTlaUuc/build/asan/bin/kudu as pid 16899
I20260928 02:12:08.798655 7085 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790561432251112-7085-0/minicluster-data/master-0/wal/instance
I20260928 02:12:08.799564 7085 external_mini_cluster.cc:1428] Setting key ecbb01b6425755a7abe1161bafefa3e4
I20260928 02:12:08.802145 16965 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:12:08.808375 16968 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:12:08.801729 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36600 (local address 127.6.235.126:41321)
0928 02:12:08.802774 (+ 1045us) server_negotiation.cc:206] Beginning negotiation
0928 02:12:08.802786 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0928 02:12:08.802827 (+ 41us) server_negotiation.cc:407] Connection header received
0928 02:12:08.804372 (+ 1545us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:12:08.804402 (+ 30us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:12:08.804863 (+ 461us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:12:08.805281 (+ 418us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:12:08.806632 (+ 1351us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:12:08.806638 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:12:08.806651 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 02:12:08.806676 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:12:08.806761 (+ 85us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 02:12:08.806778 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:12:08.806958 (+ 180us) server_negotiation.cc:1035] Waiting for connection context
0928 02:12:08.807108 (+ 150us) server_negotiation.cc:299] Negotiation successful
0928 02:12:08.807433 (+ 325us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":608,"thread_start_us":415,"threads_started":1}
I20260928 02:12:08.815904 16965 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3e9d6610e8d84d4fb0e56aea67127179: Bootstrap starting.
I20260928 02:12:08.819239 16965 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3e9d6610e8d84d4fb0e56aea67127179: Neither blocks nor log segments found. Creating new log.
I20260928 02:12:08.820993 16965 log.cc:824] T 00000000000000000000000000000000 P 3e9d6610e8d84d4fb0e56aea67127179: Log is configured to *not* fsync() on all Append() calls
I20260928 02:12:08.824167 16965 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3e9d6610e8d84d4fb0e56aea67127179: No bootstrap required, opened a new log
I20260928 02:12:08.832526 16965 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3e9d6610e8d84d4fb0e56aea67127179 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3e9d6610e8d84d4fb0e56aea67127179" member_type: VOTER last_known_addr { host: "127.6.235.126" port: 41321 } }
I20260928 02:12:08.833076 16965 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3e9d6610e8d84d4fb0e56aea67127179 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:12:08.833218 16965 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3e9d6610e8d84d4fb0e56aea67127179 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3e9d6610e8d84d4fb0e56aea67127179, State: Initialized, Role: FOLLOWER
I20260928 02:12:08.833911 16965 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3e9d6610e8d84d4fb0e56aea67127179 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3e9d6610e8d84d4fb0e56aea67127179" member_type: VOTER last_known_addr { host: "127.6.235.126" port: 41321 } }
I20260928 02:12:08.834197 16965 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3e9d6610e8d84d4fb0e56aea67127179 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 02:12:08.834383 16965 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3e9d6610e8d84d4fb0e56aea67127179 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 02:12:08.834583 16965 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3e9d6610e8d84d4fb0e56aea67127179 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:12:08.837028 16965 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3e9d6610e8d84d4fb0e56aea67127179 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3e9d6610e8d84d4fb0e56aea67127179" member_type: VOTER last_known_addr { host: "127.6.235.126" port: 41321 } }
I20260928 02:12:08.837680 16965 leader_election.cc:302] T 00000000000000000000000000000000 P 3e9d6610e8d84d4fb0e56aea67127179 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3e9d6610e8d84d4fb0e56aea67127179; no voters:
I20260928 02:12:08.838480 16965 leader_election.cc:288] T 00000000000000000000000000000000 P 3e9d6610e8d84d4fb0e56aea67127179 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 02:12:08.838593 16970 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3e9d6610e8d84d4fb0e56aea67127179 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:12:08.839481 16970 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3e9d6610e8d84d4fb0e56aea67127179 [term 1 LEADER]: Becoming Leader. State: Replica: 3e9d6610e8d84d4fb0e56aea67127179, State: Running, Role: LEADER
I20260928 02:12:08.840057 16970 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3e9d6610e8d84d4fb0e56aea67127179 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3e9d6610e8d84d4fb0e56aea67127179" member_type: VOTER last_known_addr { host: "127.6.235.126" port: 41321 } }
I20260928 02:12:08.840883 16965 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3e9d6610e8d84d4fb0e56aea67127179 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 02:12:08.843356 16972 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3e9d6610e8d84d4fb0e56aea67127179 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3e9d6610e8d84d4fb0e56aea67127179. Latest consensus state: current_term: 1 leader_uuid: "3e9d6610e8d84d4fb0e56aea67127179" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3e9d6610e8d84d4fb0e56aea67127179" member_type: VOTER last_known_addr { host: "127.6.235.126" port: 41321 } } }
I20260928 02:12:08.843516 16971 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3e9d6610e8d84d4fb0e56aea67127179 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3e9d6610e8d84d4fb0e56aea67127179" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3e9d6610e8d84d4fb0e56aea67127179" member_type: VOTER last_known_addr { host: "127.6.235.126" port: 41321 } } }
I20260928 02:12:08.843817 16972 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3e9d6610e8d84d4fb0e56aea67127179 [sys.catalog]: This master's current role is: LEADER
I20260928 02:12:08.843843 16971 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3e9d6610e8d84d4fb0e56aea67127179 [sys.catalog]: This master's current role is: LEADER
I20260928 02:12:08.845921 16978 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 02:12:08.851840 16978 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 02:12:08.863932 16978 catalog_manager.cc:1347] Generated new cluster ID: 6578b0f819e649098de677aeac519d60
I20260928 02:12:08.864070 16978 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 02:12:08.878147 16978 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 02:12:08.878646 16978 catalog_manager.cc:1504] Loading token signing keys...
I20260928 02:12:08.891306 16978 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3e9d6610e8d84d4fb0e56aea67127179: Generated new TSK 0
I20260928 02:12:08.892247 16978 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 02:12:08.905665 7085 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTlaUuc/build/asan/bin/kudu
/tmp/dist-test-taskTlaUuc/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790561432251112-7085-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790561432251112-7085-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790561432251112-7085-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790561432251112-7085-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.235.65:0
--local_ip_for_outbound_sockets=127.6.235.65
--webserver_interface=127.6.235.65
--webserver_port=0
--tserver_master_addrs=127.6.235.126:41321
--builtin_ntp_servers=127.6.235.84:38797
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260928 02:12:09.280836 16989 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:12:09.281191 16989 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:12:09.281281 16989 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:12:09.288544 16989 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:12:09.288652 16989 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:12:09.288761 16989 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.235.65
I20260928 02:12:09.298065 16989 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.235.84:38797
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790561432251112-7085-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790561432251112-7085-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.235.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790561432251112-7085-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.6.235.65
--webserver_port=0
--tserver_master_addrs=127.6.235.126:41321
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.235.65
--log_dir=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790561432251112-7085-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 00:51:59 UTC on ccycloud.quasar-axgkwn.root.comops.site
build id 4676464
ASAN enabled
I20260928 02:12:09.299325 16989 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:12:09.301064 16989 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:12:09.314513 16994 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:12:09.314594 16997 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:12:09.314446 16995 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:12:09.315135 16989 server_base.cc:1034] running on GCE node
I20260928 02:12:09.315817 16989 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:12:09.316857 16989 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:12:09.318172 16989 hybrid_clock.cc:648] HybridClock initialized: now 1790561529318109 us; error 51 us; skew 500 ppm
I20260928 02:12:09.318590 16989 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260928 02:12:09.321358 16989 webserver.cc:460] Webserver started at http://127.6.235.65:41059/ using document root <none> and password file <none>
I20260928 02:12:09.322148 16989 fs_manager.cc:359] Metadata directory not provided
I20260928 02:12:09.322289 16989 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:12:09.322724 16989 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:12:09.325970 16989 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790561432251112-7085-0/minicluster-data/ts-0/data/instance:
uuid: "3e2542c3f9f149e1801cc7c33adb455a"
format_stamp: "Formatted at 2026-09-28 02:12:09 on dist-test-slave-86gv"
server_key: "27bf4a15af70f490b3df2a7e83a25c4d"
server_key_iv: "4083cf6b35aafa1df0e5bcfbcb2a4d96"
server_key_version: "encryptionkey@0"
I20260928 02:12:09.326813 16989 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790561432251112-7085-0/minicluster-data/ts-0/wal/instance:
uuid: "3e2542c3f9f149e1801cc7c33adb455a"
format_stamp: "Formatted at 2026-09-28 02:12:09 on dist-test-slave-86gv"
server_key: "27bf4a15af70f490b3df2a7e83a25c4d"
server_key_iv: "4083cf6b35aafa1df0e5bcfbcb2a4d96"
server_key_version: "encryptionkey@0"
I20260928 02:12:09.332526 16989 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260928 02:12:09.336597 17003 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:12:09.338207 16989 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260928 02:12:09.338394 16989 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790561432251112-7085-0/minicluster-data/ts-0/data,/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790561432251112-7085-0/minicluster-data/ts-0/wal
uuid: "3e2542c3f9f149e1801cc7c33adb455a"
format_stamp: "Formatted at 2026-09-28 02:12:09 on dist-test-slave-86gv"
server_key: "27bf4a15af70f490b3df2a7e83a25c4d"
server_key_iv: "4083cf6b35aafa1df0e5bcfbcb2a4d96"
server_key_version: "encryptionkey@0"
I20260928 02:12:09.338626 16989 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790561432251112-7085-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790561432251112-7085-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790561432251112-7085-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:12:09.344951 16989 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:12:09.345744 16989 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:12:09.346006 16989 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:12:09.346956 16989 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:12:09.348371 16989 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:12:09.348480 16989 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:12:09.348598 16989 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:12:09.348691 16989 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:12:09.382179 16989 rpc_server.cc:307] RPC server started. Bound to: 127.6.235.65:43373
I20260928 02:12:09.382223 17115 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.235.65:43373 every 8 connection(s)
I20260928 02:12:09.383563 16989 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790561432251112-7085-0/minicluster-data/ts-0/data/info.pb
I20260928 02:12:09.384519 7085 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTlaUuc/build/asan/bin/kudu as pid 16989
I20260928 02:12:09.384717 7085 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790561432251112-7085-0/minicluster-data/ts-0/wal/instance
I20260928 02:12:09.385450 7085 external_mini_cluster.cc:1428] Setting key 0d95603f855adeba99f50054a9887667
I20260928 02:12:09.388671 7085 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTlaUuc/build/asan/bin/kudu
/tmp/dist-test-taskTlaUuc/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790561432251112-7085-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790561432251112-7085-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790561432251112-7085-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790561432251112-7085-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.235.66:0
--local_ip_for_outbound_sockets=127.6.235.66
--webserver_interface=127.6.235.66
--webserver_port=0
--tserver_master_addrs=127.6.235.126:41321
--builtin_ntp_servers=127.6.235.84:38797
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260928 02:12:09.393947 17118 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:12:09.386482 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.235.65:39139 (local address 127.6.235.126:41321)
0928 02:12:09.386960 (+ 478us) server_negotiation.cc:206] Beginning negotiation
0928 02:12:09.386966 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 02:12:09.388540 (+ 1574us) server_negotiation.cc:407] Connection header received
0928 02:12:09.391359 (+ 2819us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:12:09.391363 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:12:09.391419 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:12:09.391512 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:12:09.393195 (+ 1683us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:12:09.393199 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:12:09.393202 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 02:12:09.393218 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:12:09.393260 (+ 42us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 02:12:09.393272 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:12:09.393367 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0928 02:12:09.393641 (+ 274us) server_negotiation.cc:299] Negotiation successful
0928 02:12:09.393719 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":346,"thread_start_us":236,"threads_started":1}
I20260928 02:12:09.395219 17119 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:12:09.387167 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.235.126:41321 (local address 127.6.235.65:39139)
0928 02:12:09.388361 (+ 1194us) negotiation.cc:107] Waiting for socket to connect
0928 02:12:09.388422 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0928 02:12:09.391061 (+ 2639us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:12:09.391790 (+ 729us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:12:09.391804 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:12:09.391934 (+ 130us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:12:09.392795 (+ 861us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 02:12:09.392800 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:12:09.392855 (+ 55us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 02:12:09.393007 (+ 152us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:12:09.393370 (+ 363us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:12:09.393391 (+ 21us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:12:09.393401 (+ 10us) client_negotiation.cc:769] Sending connection context
0928 02:12:09.393676 (+ 275us) client_negotiation.cc:240] Negotiation successful
0928 02:12:09.393940 (+ 264us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":589,"thread_start_us":350,"threads_started":1}
I20260928 02:12:09.396807 17116 heartbeater.cc:347] Connected to a master server at 127.6.235.126:41321
I20260928 02:12:09.397174 17116 heartbeater.cc:464] Registering TS with master...
I20260928 02:12:09.397933 17116 heartbeater.cc:511] Master 127.6.235.126:41321 requested a full tablet report, sending...
I20260928 02:12:09.400120 16930 ts_manager.cc:194] Registered new tserver with Master: 3e2542c3f9f149e1801cc7c33adb455a (127.6.235.65:43373)
W20260928 02:12:09.751091 17120 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:12:09.751377 17120 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:12:09.751523 17120 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:12:09.758962 17120 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:12:09.759076 17120 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:12:09.759291 17120 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.235.66
I20260928 02:12:09.768471 17120 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.235.84:38797
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790561432251112-7085-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790561432251112-7085-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.235.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790561432251112-7085-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.6.235.66
--webserver_port=0
--tserver_master_addrs=127.6.235.126:41321
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.235.66
--log_dir=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790561432251112-7085-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 00:51:59 UTC on ccycloud.quasar-axgkwn.root.comops.site
build id 4676464
ASAN enabled
I20260928 02:12:09.769878 17120 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:12:09.771611 17120 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:12:09.786178 17126 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:12:09.786336 17128 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:12:09.786371 17125 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:12:09.787520 17120 server_base.cc:1034] running on GCE node
I20260928 02:12:09.788300 17120 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:12:09.789426 17120 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:12:09.790638 17120 hybrid_clock.cc:648] HybridClock initialized: now 1790561529790598 us; error 35 us; skew 500 ppm
I20260928 02:12:09.791064 17120 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260928 02:12:09.793713 17120 webserver.cc:460] Webserver started at http://127.6.235.66:34855/ using document root <none> and password file <none>
I20260928 02:12:09.794523 17120 fs_manager.cc:359] Metadata directory not provided
I20260928 02:12:09.794651 17120 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:12:09.795154 17120 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:12:09.799326 17120 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790561432251112-7085-0/minicluster-data/ts-1/data/instance:
uuid: "2459e7714e9e4348ae1003e985e7e4e6"
format_stamp: "Formatted at 2026-09-28 02:12:09 on dist-test-slave-86gv"
server_key: "ed0106e15f1d0096016a471b81f6f5c4"
server_key_iv: "61be867d72fcd1fcf3aa8f15c7f4fceb"
server_key_version: "encryptionkey@0"
I20260928 02:12:09.800369 17120 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790561432251112-7085-0/minicluster-data/ts-1/wal/instance:
uuid: "2459e7714e9e4348ae1003e985e7e4e6"
format_stamp: "Formatted at 2026-09-28 02:12:09 on dist-test-slave-86gv"
server_key: "ed0106e15f1d0096016a471b81f6f5c4"
server_key_iv: "61be867d72fcd1fcf3aa8f15c7f4fceb"
server_key_version: "encryptionkey@0"
I20260928 02:12:09.807538 17120 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.009s sys 0.000s
I20260928 02:12:09.812229 17134 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:12:09.814324 17120 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260928 02:12:09.814528 17120 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790561432251112-7085-0/minicluster-data/ts-1/data,/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790561432251112-7085-0/minicluster-data/ts-1/wal
uuid: "2459e7714e9e4348ae1003e985e7e4e6"
format_stamp: "Formatted at 2026-09-28 02:12:09 on dist-test-slave-86gv"
server_key: "ed0106e15f1d0096016a471b81f6f5c4"
server_key_iv: "61be867d72fcd1fcf3aa8f15c7f4fceb"
server_key_version: "encryptionkey@0"
I20260928 02:12:09.814795 17120 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790561432251112-7085-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790561432251112-7085-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790561432251112-7085-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:12:09.822619 17120 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:12:09.823436 17120 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:12:09.823676 17120 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:12:09.824637 17120 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:12:09.826143 17120 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:12:09.826241 17120 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:12:09.826318 17120 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:12:09.826381 17120 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:12:09.862515 17120 rpc_server.cc:307] RPC server started. Bound to: 127.6.235.66:40333
I20260928 02:12:09.862572 17246 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.235.66:40333 every 8 connection(s)
I20260928 02:12:09.863974 17120 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790561432251112-7085-0/minicluster-data/ts-1/data/info.pb
I20260928 02:12:09.865800 7085 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTlaUuc/build/asan/bin/kudu as pid 17120
I20260928 02:12:09.865975 7085 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790561432251112-7085-0/minicluster-data/ts-1/wal/instance
I20260928 02:12:09.866510 7085 external_mini_cluster.cc:1428] Setting key c72b2ccb75372abc2b406d31abdcdfee
I20260928 02:12:09.869532 7085 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTlaUuc/build/asan/bin/kudu
/tmp/dist-test-taskTlaUuc/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790561432251112-7085-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790561432251112-7085-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790561432251112-7085-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790561432251112-7085-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.235.67:0
--local_ip_for_outbound_sockets=127.6.235.67
--webserver_interface=127.6.235.67
--webserver_port=0
--tserver_master_addrs=127.6.235.126:41321
--builtin_ntp_servers=127.6.235.84:38797
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260928 02:12:09.873240 17118 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:12:09.867124 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.235.66:46797 (local address 127.6.235.126:41321)
0928 02:12:09.867323 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0928 02:12:09.867329 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 02:12:09.869053 (+ 1724us) server_negotiation.cc:407] Connection header received
0928 02:12:09.870831 (+ 1778us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:12:09.870836 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:12:09.870920 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:12:09.871029 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:12:09.872399 (+ 1370us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:12:09.872407 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:12:09.872414 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 02:12:09.872438 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:12:09.872485 (+ 47us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 02:12:09.872498 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:12:09.872637 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0928 02:12:09.872830 (+ 193us) server_negotiation.cc:299] Negotiation successful
0928 02:12:09.872934 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260928 02:12:09.874308 17249 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:12:09.867737 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.235.126:41321 (local address 127.6.235.66:46797)
0928 02:12:09.868853 (+ 1116us) negotiation.cc:107] Waiting for socket to connect
0928 02:12:09.868938 (+ 85us) client_negotiation.cc:174] Beginning negotiation
0928 02:12:09.870558 (+ 1620us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:12:09.871269 (+ 711us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:12:09.871278 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:12:09.871380 (+ 102us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:12:09.872108 (+ 728us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 02:12:09.872110 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:12:09.872149 (+ 39us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 02:12:09.872230 (+ 81us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:12:09.872594 (+ 364us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:12:09.872606 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:12:09.872616 (+ 10us) client_negotiation.cc:769] Sending connection context
0928 02:12:09.872927 (+ 311us) client_negotiation.cc:240] Negotiation successful
0928 02:12:09.873236 (+ 309us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":542,"thread_start_us":307,"threads_started":1}
I20260928 02:12:09.875871 17247 heartbeater.cc:347] Connected to a master server at 127.6.235.126:41321
I20260928 02:12:09.876175 17247 heartbeater.cc:464] Registering TS with master...
I20260928 02:12:09.876777 17247 heartbeater.cc:511] Master 127.6.235.126:41321 requested a full tablet report, sending...
I20260928 02:12:09.878546 16930 ts_manager.cc:194] Registered new tserver with Master: 2459e7714e9e4348ae1003e985e7e4e6 (127.6.235.66:40333)
W20260928 02:12:10.204577 17250 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:12:10.204862 17250 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:12:10.205034 17250 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:12:10.212252 17250 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:12:10.212366 17250 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:12:10.212492 17250 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.235.67
I20260928 02:12:10.221853 17250 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.235.84:38797
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790561432251112-7085-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790561432251112-7085-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.235.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790561432251112-7085-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.6.235.67
--webserver_port=0
--tserver_master_addrs=127.6.235.126:41321
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.235.67
--log_dir=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790561432251112-7085-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 00:51:59 UTC on ccycloud.quasar-axgkwn.root.comops.site
build id 4676464
ASAN enabled
I20260928 02:12:10.223201 17250 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:12:10.224962 17250 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:12:10.238492 17258 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:12:10.238452 17256 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:12:10.238520 17255 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:12:10.239435 17250 server_base.cc:1034] running on GCE node
I20260928 02:12:10.240051 17250 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:12:10.241101 17250 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:12:10.242343 17250 hybrid_clock.cc:648] HybridClock initialized: now 1790561530242288 us; error 36 us; skew 500 ppm
I20260928 02:12:10.242707 17250 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260928 02:12:10.245289 17250 webserver.cc:460] Webserver started at http://127.6.235.67:46281/ using document root <none> and password file <none>
I20260928 02:12:10.246038 17250 fs_manager.cc:359] Metadata directory not provided
I20260928 02:12:10.246179 17250 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:12:10.246531 17250 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:12:10.249498 17250 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790561432251112-7085-0/minicluster-data/ts-2/data/instance:
uuid: "6d78804ba48546569c89ea60d3631f5f"
format_stamp: "Formatted at 2026-09-28 02:12:10 on dist-test-slave-86gv"
server_key: "b0a8ab0fe843375959e727ada6dda572"
server_key_iv: "0612c3f61123f4aff1dd54136a91b26a"
server_key_version: "encryptionkey@0"
I20260928 02:12:10.250304 17250 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790561432251112-7085-0/minicluster-data/ts-2/wal/instance:
uuid: "6d78804ba48546569c89ea60d3631f5f"
format_stamp: "Formatted at 2026-09-28 02:12:10 on dist-test-slave-86gv"
server_key: "b0a8ab0fe843375959e727ada6dda572"
server_key_iv: "0612c3f61123f4aff1dd54136a91b26a"
server_key_version: "encryptionkey@0"
I20260928 02:12:10.255822 17250 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260928 02:12:10.259548 17264 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:12:10.261083 17250 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 02:12:10.261268 17250 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790561432251112-7085-0/minicluster-data/ts-2/data,/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790561432251112-7085-0/minicluster-data/ts-2/wal
uuid: "6d78804ba48546569c89ea60d3631f5f"
format_stamp: "Formatted at 2026-09-28 02:12:10 on dist-test-slave-86gv"
server_key: "b0a8ab0fe843375959e727ada6dda572"
server_key_iv: "0612c3f61123f4aff1dd54136a91b26a"
server_key_version: "encryptionkey@0"
I20260928 02:12:10.261497 17250 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790561432251112-7085-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790561432251112-7085-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790561432251112-7085-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:12:10.267343 17250 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:12:10.268218 17250 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:12:10.268503 17250 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:12:10.269501 17250 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:12:10.270959 17250 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:12:10.271044 17250 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:12:10.271204 17250 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:12:10.271287 17250 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:12:10.304178 17250 rpc_server.cc:307] RPC server started. Bound to: 127.6.235.67:33761
I20260928 02:12:10.304227 17376 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.235.67:33761 every 8 connection(s)
I20260928 02:12:10.305637 17250 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790561432251112-7085-0/minicluster-data/ts-2/data/info.pb
I20260928 02:12:10.314170 17118 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:12:10.308363 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.235.67:38333 (local address 127.6.235.126:41321)
0928 02:12:10.308550 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0928 02:12:10.308554 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:12:10.309953 (+ 1399us) server_negotiation.cc:407] Connection header received
0928 02:12:10.311814 (+ 1861us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:12:10.311820 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:12:10.311903 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:12:10.312020 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:12:10.313395 (+ 1375us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:12:10.313403 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:12:10.313410 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 02:12:10.313430 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:12:10.313473 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 02:12:10.313486 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:12:10.313613 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0928 02:12:10.313809 (+ 196us) server_negotiation.cc:299] Negotiation successful
0928 02:12:10.313897 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260928 02:12:10.314914 7085 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTlaUuc/build/asan/bin/kudu as pid 17250
I20260928 02:12:10.315105 7085 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790561432251112-7085-0/minicluster-data/ts-2/wal/instance
I20260928 02:12:10.315246 17379 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:12:10.308825 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.235.126:41321 (local address 127.6.235.67:38333)
0928 02:12:10.309788 (+ 963us) negotiation.cc:107] Waiting for socket to connect
0928 02:12:10.309823 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0928 02:12:10.311539 (+ 1716us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:12:10.312251 (+ 712us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:12:10.312260 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:12:10.312353 (+ 93us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:12:10.313087 (+ 734us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 02:12:10.313090 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:12:10.313134 (+ 44us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 02:12:10.313221 (+ 87us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:12:10.313581 (+ 360us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:12:10.313591 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:12:10.313600 (+ 9us) client_negotiation.cc:769] Sending connection context
0928 02:12:10.313883 (+ 283us) client_negotiation.cc:240] Negotiation successful
0928 02:12:10.314151 (+ 268us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":508,"thread_start_us":258,"threads_started":1}
I20260928 02:12:10.315768 7085 external_mini_cluster.cc:1428] Setting key 9a828125c2691d7373cd0d878cf78f58
I20260928 02:12:10.316795 17377 heartbeater.cc:347] Connected to a master server at 127.6.235.126:41321
I20260928 02:12:10.317217 17377 heartbeater.cc:464] Registering TS with master...
I20260928 02:12:10.317826 17377 heartbeater.cc:511] Master 127.6.235.126:41321 requested a full tablet report, sending...
I20260928 02:12:10.319203 16930 ts_manager.cc:194] Registered new tserver with Master: 6d78804ba48546569c89ea60d3631f5f (127.6.235.67:33761)
I20260928 02:12:10.331130 7085 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260928 02:12:10.337531 17118 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:12:10.335941 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36606 (local address 127.6.235.126:41321)
0928 02:12:10.336169 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0928 02:12:10.336178 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 02:12:10.336621 (+ 443us) server_negotiation.cc:407] Connection header received
0928 02:12:10.336743 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:12:10.336746 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:12:10.336801 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:12:10.336958 (+ 157us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:12:10.337282 (+ 324us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36606: BlockingRecv error: recv got EOF from 127.0.0.1:36606 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
I20260928 02:12:10.337986 7085 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskTlaUuc/build/asan/bin/kudu pid 16989
I20260928 02:12:10.341634 17386 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:12:10.339035 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34622 (local address 127.6.235.65:43373)
0928 02:12:10.339540 (+ 505us) server_negotiation.cc:206] Beginning negotiation
0928 02:12:10.339547 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 02:12:10.339572 (+ 25us) server_negotiation.cc:407] Connection header received
0928 02:12:10.339629 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:12:10.339634 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:12:10.339854 (+ 220us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:12:10.339988 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:12:10.340863 (+ 875us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:12:10.340878 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:12:10.340940 (+ 62us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 02:12:10.340968 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:12:10.341025 (+ 57us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 02:12:10.341041 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:12:10.341143 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0928 02:12:10.341242 (+ 99us) server_negotiation.cc:299] Negotiation successful
0928 02:12:10.341319 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":361,"thread_start_us":252,"threads_started":1}
I20260928 02:12:10.342949 17111 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 02:12:10.423771 17116 heartbeater.cc:503] Master 127.6.235.126:41321 was elected leader, sending a full tablet report...
I20260928 02:12:10.425967 7085 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTlaUuc/build/asan/bin/kudu with pid 16989
I20260928 02:12:10.443415 7085 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskTlaUuc/build/asan/bin/kudu pid 17120
I20260928 02:12:10.447386 17388 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:12:10.444307 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59076 (local address 127.6.235.66:40333)
0928 02:12:10.444787 (+ 480us) server_negotiation.cc:206] Beginning negotiation
0928 02:12:10.444794 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 02:12:10.444820 (+ 26us) server_negotiation.cc:407] Connection header received
0928 02:12:10.444957 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:12:10.444965 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:12:10.445187 (+ 222us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:12:10.445369 (+ 182us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:12:10.446388 (+ 1019us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:12:10.446395 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:12:10.446407 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 02:12:10.446420 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:12:10.446463 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 02:12:10.446477 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:12:10.446628 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0928 02:12:10.446727 (+ 99us) server_negotiation.cc:299] Negotiation successful
0928 02:12:10.446838 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":348,"thread_start_us":225,"threads_started":1}
I20260928 02:12:10.448632 17242 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 02:12:10.530023 7085 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTlaUuc/build/asan/bin/kudu with pid 17120
I20260928 02:12:10.547036 7085 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskTlaUuc/build/asan/bin/kudu pid 17250
I20260928 02:12:10.550697 17390 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:12:10.547924 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37020 (local address 127.6.235.67:33761)
0928 02:12:10.548553 (+ 629us) server_negotiation.cc:206] Beginning negotiation
0928 02:12:10.548560 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 02:12:10.548593 (+ 33us) server_negotiation.cc:407] Connection header received
0928 02:12:10.548654 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:12:10.548661 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:12:10.548982 (+ 321us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:12:10.549118 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:12:10.550092 (+ 974us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:12:10.550098 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:12:10.550111 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 02:12:10.550122 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:12:10.550162 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 02:12:10.550179 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:12:10.550312 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0928 02:12:10.550361 (+ 49us) server_negotiation.cc:299] Negotiation successful
0928 02:12:10.550447 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":468,"thread_start_us":307,"threads_started":1}
I20260928 02:12:10.551945 17372 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 02:12:10.633396 7085 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTlaUuc/build/asan/bin/kudu with pid 17250
I20260928 02:12:10.655251 7085 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskTlaUuc/build/asan/bin/kudu pid 16899
I20260928 02:12:10.656432 16960 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 02:12:10.758809 7085 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTlaUuc/build/asan/bin/kudu with pid 16899
2026-09-28T02:12:10Z chronyd exiting
[ OK ] SecurityITest.TestRequireEncryptionInsecureCluster (2456 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationSecureCluster
Loading random data
Initializing database '/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790561432251112-7085-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 02:12:10 dist-test-slave-86gv krb5kdc[17395](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 02:12:10 dist-test-slave-86gv krb5kdc[17395](info): set up 2 sockets
Sep 28 02:12:10 dist-test-slave-86gv krb5kdc[17395](info): commencing operation
krb5kdc: starting...
W20260928 02:12:11.874354 7085 mini_kdc.cc:121] Time spent starting KDC: real 1.073s user 0.001s sys 0.024s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790561432251112-7085-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790561432251112-7085-0/krb5kdc/test-user.keytab.
Sep 28 02:12:12 dist-test-slave-86gv krb5kdc[17395](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561532, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T02:12:12Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T02:12:12Z Disabled control of system clock
WARNING: no policy specified for kudu/127.6.235.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.235.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.235.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790561432251112-7085-0/krb5kdc/kudu_127.6.235.126.keytab.
Entry for principal kudu/127.6.235.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790561432251112-7085-0/krb5kdc/kudu_127.6.235.126.keytab.
I20260928 02:12:12.121757 7085 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTlaUuc/build/asan/bin/kudu
/tmp/dist-test-taskTlaUuc/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790561432251112-7085-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790561432251112-7085-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790561432251112-7085-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790561432251112-7085-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.235.126:38277
--webserver_interface=127.6.235.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790561432251112-7085-0/krb5kdc/kudu_127.6.235.126.keytab
--principal=kudu/127.6.235.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.235.84:36999
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.235.126:38277
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790561432251112-7085-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790561432251112-7085-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790561432251112-7085-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:12:12.453367 17408 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:12:12.453780 17408 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:12:12.453958 17408 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:12:12.461229 17408 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 02:12:12.461409 17408 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:12:12.461522 17408 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:12:12.461634 17408 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 02:12:12.461725 17408 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 02:12:12.471618 17408 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.235.84:36999
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790561432251112-7085-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790561432251112-7085-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.235.126:38277
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790561432251112-7085-0/krb5kdc/kudu_127.6.235.126.keytab
--principal=kudu/127.6.235.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.235.126:38277
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790561432251112-7085-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.235.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790561432251112-7085-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 00:51:59 UTC on ccycloud.quasar-axgkwn.root.comops.site
build id 4676464
ASAN enabled
I20260928 02:12:12.473137 17408 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:12:12.474509 17408 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:12:12.486896 17416 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:12:12.486825 17414 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:12:12.487131 17408 server_base.cc:1034] running on GCE node
W20260928 02:12:12.486923 17413 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:12:12.488214 17408 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:12:12.489713 17408 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:12:12.490922 17408 hybrid_clock.cc:648] HybridClock initialized: now 1790561532490874 us; error 32 us; skew 500 ppm
I20260928 02:12:12.491272 17408 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 02:12:12 dist-test-slave-86gv krb5kdc[17395](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561532, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.235.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:12:12.494592 17408 init.cc:377] Logged in from keytab as kudu/127.6.235.126@KRBTEST.COM (short username kudu)
I20260928 02:12:12.496069 17408 webserver.cc:460] Webserver started at http://127.6.235.126:37061/ using document root <none> and password file <none>
I20260928 02:12:12.496773 17408 fs_manager.cc:359] Metadata directory not provided
I20260928 02:12:12.496941 17408 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:12:12.497264 17408 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:12:12.500388 17408 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790561432251112-7085-0/minicluster-data/master-0/data/instance:
uuid: "31b79a4bef944137ba8e2a00893ad446"
format_stamp: "Formatted at 2026-09-28 02:12:12 on dist-test-slave-86gv"
server_key: "31633fd3fcd200ebd61fe035e62c47dc"
server_key_iv: "348b01737ccc7b548b16993333627c61"
server_key_version: "encryptionkey@0"
I20260928 02:12:12.501263 17408 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790561432251112-7085-0/minicluster-data/master-0/wal/instance:
uuid: "31b79a4bef944137ba8e2a00893ad446"
format_stamp: "Formatted at 2026-09-28 02:12:12 on dist-test-slave-86gv"
server_key: "31633fd3fcd200ebd61fe035e62c47dc"
server_key_iv: "348b01737ccc7b548b16993333627c61"
server_key_version: "encryptionkey@0"
I20260928 02:12:12.506580 17408 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260928 02:12:12.510088 17423 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:12:12.511422 17408 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260928 02:12:12.511605 17408 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790561432251112-7085-0/minicluster-data/master-0/data,/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790561432251112-7085-0/minicluster-data/master-0/wal
uuid: "31b79a4bef944137ba8e2a00893ad446"
format_stamp: "Formatted at 2026-09-28 02:12:12 on dist-test-slave-86gv"
server_key: "31633fd3fcd200ebd61fe035e62c47dc"
server_key_iv: "348b01737ccc7b548b16993333627c61"
server_key_version: "encryptionkey@0"
I20260928 02:12:12.511803 17408 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790561432251112-7085-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790561432251112-7085-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790561432251112-7085-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:12:12.533164 17408 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:12:12.537968 17408 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:12:12.538278 17408 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:12:12.557166 17408 rpc_server.cc:307] RPC server started. Bound to: 127.6.235.126:38277
I20260928 02:12:12.557178 17474 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.235.126:38277 every 8 connection(s)
I20260928 02:12:12.558668 17408 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790561432251112-7085-0/minicluster-data/master-0/data/info.pb
I20260928 02:12:12.562701 17475 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:12:12.566126 7085 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTlaUuc/build/asan/bin/kudu as pid 17408
I20260928 02:12:12.566334 7085 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790561432251112-7085-0/minicluster-data/master-0/wal/instance
I20260928 02:12:12.567117 7085 external_mini_cluster.cc:1428] Setting key 1b4915f9d6f82ac1fc35ca1fcc066df6
I20260928 02:12:12.576545 17475 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 31b79a4bef944137ba8e2a00893ad446: Bootstrap starting.
I20260928 02:12:12.580452 17475 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 31b79a4bef944137ba8e2a00893ad446: Neither blocks nor log segments found. Creating new log.
I20260928 02:12:12.581936 17475 log.cc:824] T 00000000000000000000000000000000 P 31b79a4bef944137ba8e2a00893ad446: Log is configured to *not* fsync() on all Append() calls
Sep 28 02:12:12 dist-test-slave-86gv krb5kdc[17395](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561532, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.235.126@KRBTEST.COM
I20260928 02:12:12.585292 17475 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 31b79a4bef944137ba8e2a00893ad446: No bootstrap required, opened a new log
I20260928 02:12:12.591704 17478 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:12:12.569361 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35630 (local address 127.6.235.126:38277)
0928 02:12:12.570361 (+ 1000us) server_negotiation.cc:206] Beginning negotiation
0928 02:12:12.570375 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0928 02:12:12.570419 (+ 44us) server_negotiation.cc:407] Connection header received
0928 02:12:12.572095 (+ 1676us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:12:12.572117 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:12:12.572538 (+ 421us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:12:12.572952 (+ 414us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:12:12.574693 (+ 1741us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:12:12.576006 (+ 1313us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:12:12.577971 (+ 1965us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:12:12.578705 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:12:12.578926 (+ 221us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:12:12.584260 (+ 5334us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:12:12.584268 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:12:12.584287 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:12:12.584327 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:12:12.587576 (+ 3249us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:12:12.588414 (+ 838us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:12:12.588424 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:12:12.588429 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:12:12.588590 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:12:12.589002 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:12:12.589005 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:12:12.589007 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:12:12.589539 (+ 532us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:12:12.589785 (+ 246us) server_negotiation.cc:1035] Waiting for connection context
0928 02:12:12.590222 (+ 437us) server_negotiation.cc:299] Negotiation successful
0928 02:12:12.590620 (+ 398us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":585,"thread_start_us":371,"threads_started":1}
I20260928 02:12:12.594377 17475 raft_consensus.cc:348] T 00000000000000000000000000000000 P 31b79a4bef944137ba8e2a00893ad446 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "31b79a4bef944137ba8e2a00893ad446" member_type: VOTER last_known_addr { host: "127.6.235.126" port: 38277 } }
I20260928 02:12:12.594862 17475 raft_consensus.cc:374] T 00000000000000000000000000000000 P 31b79a4bef944137ba8e2a00893ad446 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:12:12.595014 17475 raft_consensus.cc:729] T 00000000000000000000000000000000 P 31b79a4bef944137ba8e2a00893ad446 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 31b79a4bef944137ba8e2a00893ad446, State: Initialized, Role: FOLLOWER
I20260928 02:12:12.595626 17475 consensus_queue.cc:260] T 00000000000000000000000000000000 P 31b79a4bef944137ba8e2a00893ad446 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "31b79a4bef944137ba8e2a00893ad446" member_type: VOTER last_known_addr { host: "127.6.235.126" port: 38277 } }
I20260928 02:12:12.595852 17475 raft_consensus.cc:388] T 00000000000000000000000000000000 P 31b79a4bef944137ba8e2a00893ad446 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 02:12:12.595996 17475 raft_consensus.cc:482] T 00000000000000000000000000000000 P 31b79a4bef944137ba8e2a00893ad446 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 02:12:12.596132 17475 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 31b79a4bef944137ba8e2a00893ad446 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:12:12.598326 17475 raft_consensus.cc:504] T 00000000000000000000000000000000 P 31b79a4bef944137ba8e2a00893ad446 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "31b79a4bef944137ba8e2a00893ad446" member_type: VOTER last_known_addr { host: "127.6.235.126" port: 38277 } }
I20260928 02:12:12.598803 17475 leader_election.cc:302] T 00000000000000000000000000000000 P 31b79a4bef944137ba8e2a00893ad446 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 31b79a4bef944137ba8e2a00893ad446; no voters:
I20260928 02:12:12.599367 17475 leader_election.cc:288] T 00000000000000000000000000000000 P 31b79a4bef944137ba8e2a00893ad446 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 02:12:12.599514 17480 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 31b79a4bef944137ba8e2a00893ad446 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:12:12.601030 17480 raft_consensus.cc:686] T 00000000000000000000000000000000 P 31b79a4bef944137ba8e2a00893ad446 [term 1 LEADER]: Becoming Leader. State: Replica: 31b79a4bef944137ba8e2a00893ad446, State: Running, Role: LEADER
I20260928 02:12:12.601608 17480 consensus_queue.cc:237] T 00000000000000000000000000000000 P 31b79a4bef944137ba8e2a00893ad446 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "31b79a4bef944137ba8e2a00893ad446" member_type: VOTER last_known_addr { host: "127.6.235.126" port: 38277 } }
I20260928 02:12:12.601888 17475 sys_catalog.cc:564] T 00000000000000000000000000000000 P 31b79a4bef944137ba8e2a00893ad446 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 02:12:12.605182 17481 sys_catalog.cc:455] T 00000000000000000000000000000000 P 31b79a4bef944137ba8e2a00893ad446 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "31b79a4bef944137ba8e2a00893ad446" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "31b79a4bef944137ba8e2a00893ad446" member_type: VOTER last_known_addr { host: "127.6.235.126" port: 38277 } } }
I20260928 02:12:12.605569 17481 sys_catalog.cc:458] T 00000000000000000000000000000000 P 31b79a4bef944137ba8e2a00893ad446 [sys.catalog]: This master's current role is: LEADER
I20260928 02:12:12.605747 17482 sys_catalog.cc:455] T 00000000000000000000000000000000 P 31b79a4bef944137ba8e2a00893ad446 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 31b79a4bef944137ba8e2a00893ad446. Latest consensus state: current_term: 1 leader_uuid: "31b79a4bef944137ba8e2a00893ad446" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "31b79a4bef944137ba8e2a00893ad446" member_type: VOTER last_known_addr { host: "127.6.235.126" port: 38277 } } }
I20260928 02:12:12.606048 17482 sys_catalog.cc:458] T 00000000000000000000000000000000 P 31b79a4bef944137ba8e2a00893ad446 [sys.catalog]: This master's current role is: LEADER
I20260928 02:12:12.608374 17487 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 02:12:12.613919 17487 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 02:12:12.622315 17487 catalog_manager.cc:1347] Generated new cluster ID: 4ac137767d06433182605cd3c14d4352
I20260928 02:12:12.622432 17487 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 02:12:12.684769 17487 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 02:12:12.686764 17487 catalog_manager.cc:1504] Loading token signing keys...
I20260928 02:12:12.693696 17487 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 31b79a4bef944137ba8e2a00893ad446: Generated new TSK 0
I20260928 02:12:12.694810 17487 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.6.235.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.235.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.235.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790561432251112-7085-0/krb5kdc/kudu_127.6.235.65.keytab.
Entry for principal kudu/127.6.235.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790561432251112-7085-0/krb5kdc/kudu_127.6.235.65.keytab.
I20260928 02:12:12.764482 7085 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTlaUuc/build/asan/bin/kudu
/tmp/dist-test-taskTlaUuc/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790561432251112-7085-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790561432251112-7085-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790561432251112-7085-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790561432251112-7085-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.235.65:0
--local_ip_for_outbound_sockets=127.6.235.65
--webserver_interface=127.6.235.65
--webserver_port=0
--tserver_master_addrs=127.6.235.126:38277
--keytab_file=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790561432251112-7085-0/krb5kdc/kudu_127.6.235.65.keytab
--principal=kudu/127.6.235.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.235.84:36999
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790561432251112-7085-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790561432251112-7085-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790561432251112-7085-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:12:13.104547 17501 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:12:13.104825 17501 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:12:13.105067 17501 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:12:13.112591 17501 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:12:13.112706 17501 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:12:13.112963 17501 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.235.65
I20260928 02:12:13.123435 17501 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.235.84:36999
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790561432251112-7085-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790561432251112-7085-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790561432251112-7085-0/krb5kdc/kudu_127.6.235.65.keytab
--principal=kudu/127.6.235.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.235.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790561432251112-7085-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.6.235.65
--webserver_port=0
--tserver_master_addrs=127.6.235.126:38277
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.235.65
--log_dir=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790561432251112-7085-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 00:51:59 UTC on ccycloud.quasar-axgkwn.root.comops.site
build id 4676464
ASAN enabled
I20260928 02:12:13.124876 17501 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:12:13.126312 17501 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:12:13.140108 17507 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:12:13.140153 17506 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:12:13.140072 17509 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:12:13.146276 17501 server_base.cc:1034] running on GCE node
I20260928 02:12:13.147045 17501 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:12:13.148085 17501 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:12:13.149320 17501 hybrid_clock.cc:648] HybridClock initialized: now 1790561533149266 us; error 50 us; skew 500 ppm
I20260928 02:12:13.149679 17501 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 02:12:13 dist-test-slave-86gv krb5kdc[17395](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561533, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.235.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:12:13.152824 17501 init.cc:377] Logged in from keytab as kudu/127.6.235.65@KRBTEST.COM (short username kudu)
I20260928 02:12:13.154385 17501 webserver.cc:460] Webserver started at http://127.6.235.65:45757/ using document root <none> and password file <none>
I20260928 02:12:13.155117 17501 fs_manager.cc:359] Metadata directory not provided
I20260928 02:12:13.155260 17501 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:12:13.155571 17501 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:12:13.158629 17501 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790561432251112-7085-0/minicluster-data/ts-0/data/instance:
uuid: "327bc76b8ba447fb821d916ca281b6a5"
format_stamp: "Formatted at 2026-09-28 02:12:13 on dist-test-slave-86gv"
server_key: "a5b9fbebf29ce0f92ac6cd2253356739"
server_key_iv: "ae315ba1ee78d0258f37a674af8c37e9"
server_key_version: "encryptionkey@0"
I20260928 02:12:13.159603 17501 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790561432251112-7085-0/minicluster-data/ts-0/wal/instance:
uuid: "327bc76b8ba447fb821d916ca281b6a5"
format_stamp: "Formatted at 2026-09-28 02:12:13 on dist-test-slave-86gv"
server_key: "a5b9fbebf29ce0f92ac6cd2253356739"
server_key_iv: "ae315ba1ee78d0258f37a674af8c37e9"
server_key_version: "encryptionkey@0"
I20260928 02:12:13.164845 17501 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260928 02:12:13.168339 17516 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:12:13.169718 17501 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 02:12:13.169896 17501 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790561432251112-7085-0/minicluster-data/ts-0/data,/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790561432251112-7085-0/minicluster-data/ts-0/wal
uuid: "327bc76b8ba447fb821d916ca281b6a5"
format_stamp: "Formatted at 2026-09-28 02:12:13 on dist-test-slave-86gv"
server_key: "a5b9fbebf29ce0f92ac6cd2253356739"
server_key_iv: "ae315ba1ee78d0258f37a674af8c37e9"
server_key_version: "encryptionkey@0"
I20260928 02:12:13.170101 17501 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790561432251112-7085-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790561432251112-7085-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790561432251112-7085-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:12:13.195326 17501 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:12:13.199697 17501 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:12:13.200001 17501 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:12:13.200997 17501 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:12:13.202453 17501 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:12:13.202557 17501 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:12:13.202703 17501 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:12:13.202773 17501 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:12:13.236349 17501 rpc_server.cc:307] RPC server started. Bound to: 127.6.235.65:34811
I20260928 02:12:13.236443 17628 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.235.65:34811 every 8 connection(s)
I20260928 02:12:13.237788 17501 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790561432251112-7085-0/minicluster-data/ts-0/data/info.pb
I20260928 02:12:13.238919 7085 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTlaUuc/build/asan/bin/kudu as pid 17501
I20260928 02:12:13.239121 7085 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790561432251112-7085-0/minicluster-data/ts-0/wal/instance
I20260928 02:12:13.239792 7085 external_mini_cluster.cc:1428] Setting key 8f93d1c1d8b6cad300ece708791f4d13
Sep 28 02:12:13 dist-test-slave-86gv krb5kdc[17395](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561533, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.235.65@KRBTEST.COM for kudu/127.6.235.126@KRBTEST.COM
I20260928 02:12:13.259730 17631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:12:13.240859 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.235.65:35107 (local address 127.6.235.126:38277)
0928 02:12:13.241366 (+ 507us) server_negotiation.cc:206] Beginning negotiation
0928 02:12:13.241370 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:12:13.242594 (+ 1224us) server_negotiation.cc:407] Connection header received
0928 02:12:13.244434 (+ 1840us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:12:13.244442 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:12:13.244538 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:12:13.244656 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:12:13.246265 (+ 1609us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:12:13.247472 (+ 1207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:12:13.249195 (+ 1723us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:12:13.249632 (+ 437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:12:13.249782 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:12:13.253790 (+ 4008us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:12:13.253797 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:12:13.253802 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:12:13.253844 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:12:13.256418 (+ 2574us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:12:13.257267 (+ 849us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:12:13.257271 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:12:13.257273 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:12:13.257401 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:12:13.257848 (+ 447us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:12:13.257857 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:12:13.257859 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:12:13.258120 (+ 261us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:12:13.258335 (+ 215us) server_negotiation.cc:1035] Waiting for connection context
0928 02:12:13.259096 (+ 761us) server_negotiation.cc:299] Negotiation successful
0928 02:12:13.259341 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":407,"thread_start_us":281,"threads_started":1}
I20260928 02:12:13.260775 17632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:12:13.241512 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.235.126:38277 (local address 127.6.235.65:35107)
0928 02:12:13.242408 (+ 896us) negotiation.cc:107] Waiting for socket to connect
0928 02:12:13.242468 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0928 02:12:13.244152 (+ 1684us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:12:13.244946 (+ 794us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:12:13.244960 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:12:13.245493 (+ 533us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:12:13.246079 (+ 586us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:12:13.246097 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:12:13.247628 (+ 1531us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:12:13.247631 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:12:13.249035 (+ 1404us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:12:13.249047 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:12:13.249884 (+ 837us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:12:13.249892 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:12:13.250136 (+ 244us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:12:13.250879 (+ 743us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:12:13.250919 (+ 40us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:12:13.253536 (+ 2617us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:12:13.256567 (+ 3031us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:12:13.256577 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:12:13.256597 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:12:13.257126 (+ 529us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:12:13.257513 (+ 387us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:12:13.257520 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:12:13.257524 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:12:13.257720 (+ 196us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:12:13.258318 (+ 598us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:12:13.258326 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:12:13.258691 (+ 365us) client_negotiation.cc:769] Sending connection context
0928 02:12:13.258978 (+ 287us) client_negotiation.cc:240] Negotiation successful
0928 02:12:13.259548 (+ 570us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":603,"thread_start_us":384,"threads_started":1}
I20260928 02:12:13.262818 17629 heartbeater.cc:347] Connected to a master server at 127.6.235.126:38277
WARNING: no policy specified for kudu/127.6.235.66@KRBTEST.COM; defaulting to no policy
I20260928 02:12:13.263144 17629 heartbeater.cc:464] Registering TS with master...
I20260928 02:12:13.263929 17629 heartbeater.cc:511] Master 127.6.235.126:38277 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.235.66@KRBTEST.COM" created.
I20260928 02:12:13.266492 17440 ts_manager.cc:194] Registered new tserver with Master: 327bc76b8ba447fb821d916ca281b6a5 (127.6.235.65:34811)
I20260928 02:12:13.267990 17440 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.235.65@KRBTEST.COM'} at 127.6.235.65:35107
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.235.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790561432251112-7085-0/krb5kdc/kudu_127.6.235.66.keytab.
Entry for principal kudu/127.6.235.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790561432251112-7085-0/krb5kdc/kudu_127.6.235.66.keytab.
I20260928 02:12:13.297672 7085 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTlaUuc/build/asan/bin/kudu
/tmp/dist-test-taskTlaUuc/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790561432251112-7085-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790561432251112-7085-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790561432251112-7085-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790561432251112-7085-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.235.66:0
--local_ip_for_outbound_sockets=127.6.235.66
--webserver_interface=127.6.235.66
--webserver_port=0
--tserver_master_addrs=127.6.235.126:38277
--keytab_file=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790561432251112-7085-0/krb5kdc/kudu_127.6.235.66.keytab
--principal=kudu/127.6.235.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.235.84:36999
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790561432251112-7085-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790561432251112-7085-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790561432251112-7085-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:12:13.637578 17635 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:12:13.637887 17635 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:12:13.637984 17635 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:12:13.645504 17635 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:12:13.645622 17635 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:12:13.645742 17635 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.235.66
I20260928 02:12:13.655642 17635 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.235.84:36999
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790561432251112-7085-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790561432251112-7085-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790561432251112-7085-0/krb5kdc/kudu_127.6.235.66.keytab
--principal=kudu/127.6.235.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.235.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790561432251112-7085-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.6.235.66
--webserver_port=0
--tserver_master_addrs=127.6.235.126:38277
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.235.66
--log_dir=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790561432251112-7085-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 00:51:59 UTC on ccycloud.quasar-axgkwn.root.comops.site
build id 4676464
ASAN enabled
I20260928 02:12:13.656970 17635 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:12:13.658398 17635 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:12:13.673081 17641 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:12:13.673099 17643 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:12:13.673130 17640 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:12:13.674396 17635 server_base.cc:1034] running on GCE node
I20260928 02:12:13.675129 17635 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:12:13.676175 17635 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:12:13.677405 17635 hybrid_clock.cc:648] HybridClock initialized: now 1790561533677357 us; error 50 us; skew 500 ppm
I20260928 02:12:13.677780 17635 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 02:12:13 dist-test-slave-86gv krb5kdc[17395](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561533, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.235.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:12:13.681653 17635 init.cc:377] Logged in from keytab as kudu/127.6.235.66@KRBTEST.COM (short username kudu)
I20260928 02:12:13.683244 17635 webserver.cc:460] Webserver started at http://127.6.235.66:38535/ using document root <none> and password file <none>
I20260928 02:12:13.683956 17635 fs_manager.cc:359] Metadata directory not provided
I20260928 02:12:13.684060 17635 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:12:13.684315 17635 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:12:13.687346 17635 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790561432251112-7085-0/minicluster-data/ts-1/data/instance:
uuid: "4d8ac0bbb79f46acaa70becb4a39edc3"
format_stamp: "Formatted at 2026-09-28 02:12:13 on dist-test-slave-86gv"
server_key: "7c800efd3f5c20e3e356511ab3d827fd"
server_key_iv: "dfcbeb59b0e74b23b272ad000f845259"
server_key_version: "encryptionkey@0"
I20260928 02:12:13.688087 17635 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790561432251112-7085-0/minicluster-data/ts-1/wal/instance:
uuid: "4d8ac0bbb79f46acaa70becb4a39edc3"
format_stamp: "Formatted at 2026-09-28 02:12:13 on dist-test-slave-86gv"
server_key: "7c800efd3f5c20e3e356511ab3d827fd"
server_key_iv: "dfcbeb59b0e74b23b272ad000f845259"
server_key_version: "encryptionkey@0"
I20260928 02:12:13.693308 17635 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.002s
I20260928 02:12:13.696944 17650 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:12:13.698514 17635 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260928 02:12:13.698652 17635 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790561432251112-7085-0/minicluster-data/ts-1/data,/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790561432251112-7085-0/minicluster-data/ts-1/wal
uuid: "4d8ac0bbb79f46acaa70becb4a39edc3"
format_stamp: "Formatted at 2026-09-28 02:12:13 on dist-test-slave-86gv"
server_key: "7c800efd3f5c20e3e356511ab3d827fd"
server_key_iv: "dfcbeb59b0e74b23b272ad000f845259"
server_key_version: "encryptionkey@0"
I20260928 02:12:13.698827 17635 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790561432251112-7085-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790561432251112-7085-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790561432251112-7085-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:12:13.710795 17635 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:12:13.715056 17635 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:12:13.715377 17635 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:12:13.716292 17635 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:12:13.717788 17635 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:12:13.717902 17635 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:12:13.718057 17635 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:12:13.718137 17635 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:12:13.753129 17635 rpc_server.cc:307] RPC server started. Bound to: 127.6.235.66:39425
I20260928 02:12:13.753176 17762 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.235.66:39425 every 8 connection(s)
I20260928 02:12:13.754531 17635 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790561432251112-7085-0/minicluster-data/ts-1/data/info.pb
I20260928 02:12:13.762537 7085 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTlaUuc/build/asan/bin/kudu as pid 17635
I20260928 02:12:13.762712 7085 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790561432251112-7085-0/minicluster-data/ts-1/wal/instance
I20260928 02:12:13.763186 7085 external_mini_cluster.cc:1428] Setting key 56aa24d715760ac9c97c7b3099f20dd7
Sep 28 02:12:13 dist-test-slave-86gv krb5kdc[17395](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561533, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.235.66@KRBTEST.COM for kudu/127.6.235.126@KRBTEST.COM
I20260928 02:12:13.775833 17631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:12:13.757252 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.235.66:42611 (local address 127.6.235.126:38277)
0928 02:12:13.757469 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0928 02:12:13.757475 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 02:12:13.758669 (+ 1194us) server_negotiation.cc:407] Connection header received
0928 02:12:13.760365 (+ 1696us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:12:13.760369 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:12:13.760435 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:12:13.760521 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:12:13.762204 (+ 1683us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:12:13.763023 (+ 819us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:12:13.764968 (+ 1945us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:12:13.765459 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:12:13.765556 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:12:13.769879 (+ 4323us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:12:13.769885 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:12:13.769888 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:12:13.769922 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:12:13.772449 (+ 2527us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:12:13.773317 (+ 868us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:12:13.773322 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:12:13.773325 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:12:13.773467 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:12:13.773973 (+ 506us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:12:13.773985 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:12:13.773988 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:12:13.774256 (+ 268us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:12:13.774436 (+ 180us) server_negotiation.cc:1035] Waiting for connection context
0928 02:12:13.775245 (+ 809us) server_negotiation.cc:299] Negotiation successful
0928 02:12:13.775489 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260928 02:12:13.776940 17765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:12:13.757820 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.235.126:38277 (local address 127.6.235.66:42611)
0928 02:12:13.758510 (+ 690us) negotiation.cc:107] Waiting for socket to connect
0928 02:12:13.758555 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0928 02:12:13.760118 (+ 1563us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:12:13.760727 (+ 609us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:12:13.760735 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:12:13.761407 (+ 672us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:12:13.762056 (+ 649us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:12:13.762074 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:12:13.763170 (+ 1096us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:12:13.763178 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:12:13.764805 (+ 1627us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:12:13.764822 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:12:13.765684 (+ 862us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:12:13.765694 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:12:13.766005 (+ 311us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:12:13.766757 (+ 752us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:12:13.766793 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:12:13.769621 (+ 2828us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:12:13.772609 (+ 2988us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:12:13.772619 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:12:13.772638 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:12:13.773177 (+ 539us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:12:13.773590 (+ 413us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:12:13.773595 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:12:13.773598 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:12:13.773845 (+ 247us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:12:13.774388 (+ 543us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:12:13.774396 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:12:13.774825 (+ 429us) client_negotiation.cc:769] Sending connection context
0928 02:12:13.775140 (+ 315us) client_negotiation.cc:240] Negotiation successful
0928 02:12:13.775715 (+ 575us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":458,"thread_start_us":258,"threads_started":1}
I20260928 02:12:13.778537 17763 heartbeater.cc:347] Connected to a master server at 127.6.235.126:38277
I20260928 02:12:13.778867 17763 heartbeater.cc:464] Registering TS with master...
I20260928 02:12:13.779626 17763 heartbeater.cc:511] Master 127.6.235.126:38277 requested a full tablet report, sending...
I20260928 02:12:13.781160 17440 ts_manager.cc:194] Registered new tserver with Master: 4d8ac0bbb79f46acaa70becb4a39edc3 (127.6.235.66:39425)
I20260928 02:12:13.781961 17440 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.235.66@KRBTEST.COM'} at 127.6.235.66:42611
WARNING: no policy specified for kudu/127.6.235.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.235.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.235.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790561432251112-7085-0/krb5kdc/kudu_127.6.235.67.keytab.
Entry for principal kudu/127.6.235.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790561432251112-7085-0/krb5kdc/kudu_127.6.235.67.keytab.
I20260928 02:12:13.818454 7085 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTlaUuc/build/asan/bin/kudu
/tmp/dist-test-taskTlaUuc/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790561432251112-7085-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790561432251112-7085-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790561432251112-7085-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790561432251112-7085-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.235.67:0
--local_ip_for_outbound_sockets=127.6.235.67
--webserver_interface=127.6.235.67
--webserver_port=0
--tserver_master_addrs=127.6.235.126:38277
--keytab_file=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790561432251112-7085-0/krb5kdc/kudu_127.6.235.67.keytab
--principal=kudu/127.6.235.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.235.84:36999
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790561432251112-7085-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790561432251112-7085-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790561432251112-7085-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:12:14.154532 17768 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:12:14.154835 17768 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:12:14.155004 17768 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:12:14.162429 17768 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:12:14.162541 17768 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:12:14.162738 17768 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.235.67
I20260928 02:12:14.173954 17768 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.235.84:36999
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790561432251112-7085-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790561432251112-7085-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790561432251112-7085-0/krb5kdc/kudu_127.6.235.67.keytab
--principal=kudu/127.6.235.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.235.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790561432251112-7085-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.6.235.67
--webserver_port=0
--tserver_master_addrs=127.6.235.126:38277
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.235.67
--log_dir=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790561432251112-7085-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 00:51:59 UTC on ccycloud.quasar-axgkwn.root.comops.site
build id 4676464
ASAN enabled
I20260928 02:12:14.175449 17768 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:12:14.176872 17768 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:12:14.192319 17773 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:12:14.192391 17774 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:12:14.192698 17776 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:12:14.193398 17768 server_base.cc:1034] running on GCE node
I20260928 02:12:14.194108 17768 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:12:14.195099 17768 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:12:14.196336 17768 hybrid_clock.cc:648] HybridClock initialized: now 1790561534196274 us; error 57 us; skew 500 ppm
I20260928 02:12:14.196684 17768 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 02:12:14 dist-test-slave-86gv krb5kdc[17395](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561534, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.235.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:12:14.200747 17768 init.cc:377] Logged in from keytab as kudu/127.6.235.67@KRBTEST.COM (short username kudu)
I20260928 02:12:14.202535 17768 webserver.cc:460] Webserver started at http://127.6.235.67:46309/ using document root <none> and password file <none>
I20260928 02:12:14.203413 17768 fs_manager.cc:359] Metadata directory not provided
I20260928 02:12:14.203554 17768 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:12:14.203989 17768 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:12:14.207965 17768 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790561432251112-7085-0/minicluster-data/ts-2/data/instance:
uuid: "add5f7c598324480835f9fdc87340fc7"
format_stamp: "Formatted at 2026-09-28 02:12:14 on dist-test-slave-86gv"
server_key: "23f8a705952c32415fc00fad47077c23"
server_key_iv: "dd6cdd5285c378961ff6163d72965cd4"
server_key_version: "encryptionkey@0"
I20260928 02:12:14.208974 17768 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790561432251112-7085-0/minicluster-data/ts-2/wal/instance:
uuid: "add5f7c598324480835f9fdc87340fc7"
format_stamp: "Formatted at 2026-09-28 02:12:14 on dist-test-slave-86gv"
server_key: "23f8a705952c32415fc00fad47077c23"
server_key_iv: "dd6cdd5285c378961ff6163d72965cd4"
server_key_version: "encryptionkey@0"
I20260928 02:12:14.214725 17768 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260928 02:12:14.218477 17783 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:12:14.219900 17768 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260928 02:12:14.220077 17768 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790561432251112-7085-0/minicluster-data/ts-2/data,/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790561432251112-7085-0/minicluster-data/ts-2/wal
uuid: "add5f7c598324480835f9fdc87340fc7"
format_stamp: "Formatted at 2026-09-28 02:12:14 on dist-test-slave-86gv"
server_key: "23f8a705952c32415fc00fad47077c23"
server_key_iv: "dd6cdd5285c378961ff6163d72965cd4"
server_key_version: "encryptionkey@0"
I20260928 02:12:14.220290 17768 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790561432251112-7085-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790561432251112-7085-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790561432251112-7085-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:12:14.245929 17768 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:12:14.250484 17768 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:12:14.250799 17768 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:12:14.251828 17768 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:12:14.253432 17768 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:12:14.253538 17768 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:12:14.253685 17768 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:12:14.253765 17768 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:12:14.271360 17629 heartbeater.cc:503] Master 127.6.235.126:38277 was elected leader, sending a full tablet report...
I20260928 02:12:14.290407 17768 rpc_server.cc:307] RPC server started. Bound to: 127.6.235.67:44873
I20260928 02:12:14.290458 17895 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.235.67:44873 every 8 connection(s)
I20260928 02:12:14.291849 17768 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790561432251112-7085-0/minicluster-data/ts-2/data/info.pb
I20260928 02:12:14.294037 7085 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTlaUuc/build/asan/bin/kudu as pid 17768
I20260928 02:12:14.294242 7085 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790561432251112-7085-0/minicluster-data/ts-2/wal/instance
I20260928 02:12:14.294739 7085 external_mini_cluster.cc:1428] Setting key 09d28d2fbf06186b75ea25876d2d5609
Sep 28 02:12:14 dist-test-slave-86gv krb5kdc[17395](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561534, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.235.67@KRBTEST.COM for kudu/127.6.235.126@KRBTEST.COM
I20260928 02:12:14.313926 17898 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:12:14.294953 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.235.67:47465 (local address 127.6.235.126:38277)
0928 02:12:14.295484 (+ 531us) server_negotiation.cc:206] Beginning negotiation
0928 02:12:14.295494 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 02:12:14.296932 (+ 1438us) server_negotiation.cc:407] Connection header received
0928 02:12:14.299017 (+ 2085us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:12:14.299023 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:12:14.299110 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:12:14.299222 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:12:14.300790 (+ 1568us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:12:14.301692 (+ 902us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:12:14.303529 (+ 1837us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:12:14.304081 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:12:14.304206 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:12:14.307644 (+ 3438us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:12:14.307649 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:12:14.307652 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:12:14.307686 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:12:14.311113 (+ 3427us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:12:14.311820 (+ 707us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:12:14.311824 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:12:14.311826 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:12:14.311959 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:12:14.312337 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:12:14.312340 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:12:14.312342 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:12:14.312610 (+ 268us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:12:14.312724 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0928 02:12:14.313496 (+ 772us) server_negotiation.cc:299] Negotiation successful
0928 02:12:14.313661 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":387,"thread_start_us":255,"threads_started":1}
I20260928 02:12:14.315059 17899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:12:14.295697 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.235.126:38277 (local address 127.6.235.67:47465)
0928 02:12:14.296735 (+ 1038us) negotiation.cc:107] Waiting for socket to connect
0928 02:12:14.296789 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0928 02:12:14.298663 (+ 1874us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:12:14.299453 (+ 790us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:12:14.299461 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:12:14.299964 (+ 503us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:12:14.300551 (+ 587us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:12:14.300568 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:12:14.301823 (+ 1255us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:12:14.301827 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:12:14.303327 (+ 1500us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:12:14.303387 (+ 60us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:12:14.304305 (+ 918us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:12:14.304310 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:12:14.304534 (+ 224us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:12:14.305232 (+ 698us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:12:14.305254 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:12:14.307440 (+ 2186us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:12:14.311240 (+ 3800us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:12:14.311247 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:12:14.311262 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:12:14.311717 (+ 455us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:12:14.312055 (+ 338us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:12:14.312058 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:12:14.312060 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:12:14.312243 (+ 183us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:12:14.312731 (+ 488us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:12:14.312737 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:12:14.313130 (+ 393us) client_negotiation.cc:769] Sending connection context
0928 02:12:14.313418 (+ 288us) client_negotiation.cc:240] Negotiation successful
0928 02:12:14.313835 (+ 417us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":752,"thread_start_us":529,"threads_started":1}
I20260928 02:12:14.316408 17896 heartbeater.cc:347] Connected to a master server at 127.6.235.126:38277
I20260928 02:12:14.316728 17896 heartbeater.cc:464] Registering TS with master...
I20260928 02:12:14.317530 17896 heartbeater.cc:511] Master 127.6.235.126:38277 requested a full tablet report, sending...
I20260928 02:12:14.318979 17440 ts_manager.cc:194] Registered new tserver with Master: add5f7c598324480835f9fdc87340fc7 (127.6.235.67:44873)
I20260928 02:12:14.319877 17440 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.235.67@KRBTEST.COM'} at 127.6.235.67:47465
I20260928 02:12:14.322937 7085 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 02:12:14.340687 17898 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:12:14.328050 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35640 (local address 127.6.235.126:38277)
0928 02:12:14.328238 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0928 02:12:14.328242 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:12:14.328665 (+ 423us) server_negotiation.cc:407] Connection header received
0928 02:12:14.328802 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:12:14.328806 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:12:14.328962 (+ 156us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:12:14.329050 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:12:14.330452 (+ 1402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:12:14.331158 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:12:14.332326 (+ 1168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:12:14.332737 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:12:14.332829 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:12:14.335225 (+ 2396us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:12:14.335233 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:12:14.335237 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:12:14.335268 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:12:14.338266 (+ 2998us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:12:14.338929 (+ 663us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:12:14.338934 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:12:14.338936 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:12:14.339060 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:12:14.339389 (+ 329us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:12:14.339392 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:12:14.339394 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:12:14.339803 (+ 409us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:12:14.339983 (+ 180us) server_negotiation.cc:1035] Waiting for connection context
0928 02:12:14.340279 (+ 296us) server_negotiation.cc:299] Negotiation successful
0928 02:12:14.340455 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260928 02:12:14.345059 17440 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35640:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 02:12:14.351923 17440 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 02:12:14.387140 17912 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:12:14.365025 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38712 (local address 127.6.235.67:44873)
0928 02:12:14.376059 (+ 11034us) server_negotiation.cc:206] Beginning negotiation
0928 02:12:14.376069 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 02:12:14.378217 (+ 2148us) server_negotiation.cc:407] Connection header received
0928 02:12:14.378352 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:12:14.378364 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:12:14.378680 (+ 316us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:12:14.378932 (+ 252us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:12:14.380055 (+ 1123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:12:14.382187 (+ 2132us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:12:14.384454 (+ 2267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:12:14.385946 (+ 1492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:12:14.386076 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:12:14.386304 (+ 228us) server_negotiation.cc:1035] Waiting for connection context
0928 02:12:14.386627 (+ 323us) server_negotiation.cc:299] Negotiation successful
0928 02:12:14.386795 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10863,"thread_start_us":10703,"threads_started":1}
I20260928 02:12:14.387209 17909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:12:14.364251 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.235.67:44873 (local address 127.0.0.1:38712)
0928 02:12:14.377871 (+ 13620us) negotiation.cc:107] Waiting for socket to connect
0928 02:12:14.377890 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0928 02:12:14.378032 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:12:14.379283 (+ 1251us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:12:14.379295 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:12:14.379331 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:12:14.379780 (+ 449us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:12:14.379794 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:12:14.382336 (+ 2542us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:12:14.382341 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:12:14.384284 (+ 1943us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:12:14.384296 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:12:14.386156 (+ 1860us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:12:14.386163 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:12:14.386356 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:12:14.386415 (+ 59us) client_negotiation.cc:769] Sending connection context
0928 02:12:14.386563 (+ 148us) client_negotiation.cc:240] Negotiation successful
0928 02:12:14.386750 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13480,"mutex_wait_us":143,"thread_start_us":10556,"threads_started":1}
I20260928 02:12:14.388041 17914 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:12:14.377894 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.235.66:39425 (local address 127.0.0.1:59922)
0928 02:12:14.380421 (+ 2527us) negotiation.cc:107] Waiting for socket to connect
0928 02:12:14.380440 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0928 02:12:14.380555 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:12:14.381170 (+ 615us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:12:14.381175 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:12:14.381199 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:12:14.381644 (+ 445us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:12:14.381655 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:12:14.383325 (+ 1670us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:12:14.383331 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:12:14.385285 (+ 1954us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:12:14.385295 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:12:14.387177 (+ 1882us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:12:14.387185 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:12:14.387369 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:12:14.387402 (+ 33us) client_negotiation.cc:769] Sending connection context
0928 02:12:14.387494 (+ 92us) client_negotiation.cc:240] Negotiation successful
0928 02:12:14.387627 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2382,"mutex_wait_us":1,"thread_start_us":2185,"threads_started":1}
I20260928 02:12:14.388258 17913 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:12:14.376161 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59922 (local address 127.6.235.66:39425)
0928 02:12:14.379892 (+ 3731us) server_negotiation.cc:206] Beginning negotiation
0928 02:12:14.379905 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0928 02:12:14.380547 (+ 642us) server_negotiation.cc:407] Connection header received
0928 02:12:14.380710 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:12:14.380716 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:12:14.380972 (+ 256us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:12:14.381188 (+ 216us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:12:14.381825 (+ 637us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:12:14.383160 (+ 1335us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:12:14.385451 (+ 2291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:12:14.386720 (+ 1269us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:12:14.386844 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:12:14.387034 (+ 190us) server_negotiation.cc:1035] Waiting for connection context
0928 02:12:14.387731 (+ 697us) server_negotiation.cc:299] Negotiation successful
0928 02:12:14.387920 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3512,"thread_start_us":3337,"threads_started":1}
I20260928 02:12:14.390374 17911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:12:14.364003 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.235.65:34811 (local address 127.0.0.1:45762)
0928 02:12:14.375254 (+ 11251us) negotiation.cc:107] Waiting for socket to connect
0928 02:12:14.375305 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0928 02:12:14.375494 (+ 189us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:12:14.378997 (+ 3503us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:12:14.379010 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:12:14.379053 (+ 43us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:12:14.380310 (+ 1257us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:12:14.380323 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:12:14.382199 (+ 1876us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:12:14.382204 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:12:14.384596 (+ 2392us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:12:14.384611 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:12:14.389515 (+ 4904us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:12:14.389521 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:12:14.389689 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:12:14.389722 (+ 33us) client_negotiation.cc:769] Sending connection context
0928 02:12:14.389848 (+ 126us) client_negotiation.cc:240] Negotiation successful
0928 02:12:14.389993 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10984,"mutex_wait_us":3,"thread_start_us":13575,"threads_started":1}
I20260928 02:12:14.390403 17910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:12:14.364265 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45762 (local address 127.6.235.65:34811)
0928 02:12:14.377733 (+ 13468us) server_negotiation.cc:206] Beginning negotiation
0928 02:12:14.377746 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0928 02:12:14.377796 (+ 50us) server_negotiation.cc:407] Connection header received
0928 02:12:14.377943 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:12:14.377956 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:12:14.378349 (+ 393us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:12:14.378608 (+ 259us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:12:14.380512 (+ 1904us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:12:14.382031 (+ 1519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:12:14.384772 (+ 2741us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:12:14.387086 (+ 2314us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:12:14.389477 (+ 2391us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:12:14.389714 (+ 237us) server_negotiation.cc:1035] Waiting for connection context
0928 02:12:14.389920 (+ 206us) server_negotiation.cc:299] Negotiation successful
0928 02:12:14.390082 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13226,"thread_start_us":15474,"threads_started":1}
I20260928 02:12:14.391706 17831 tablet_service.cc:1467] Processing CreateTablet for tablet f1ffe4cb8bf94360892a42ebb14002c8 (DEFAULT_TABLE table=test-table [id=cb72e5ad77b748509a50fe07177da74b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:12:14.391762 17698 tablet_service.cc:1467] Processing CreateTablet for tablet f1ffe4cb8bf94360892a42ebb14002c8 (DEFAULT_TABLE table=test-table [id=cb72e5ad77b748509a50fe07177da74b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:12:14.393309 17831 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f1ffe4cb8bf94360892a42ebb14002c8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:12:14.393312 17698 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f1ffe4cb8bf94360892a42ebb14002c8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:12:14.393576 17564 tablet_service.cc:1467] Processing CreateTablet for tablet f1ffe4cb8bf94360892a42ebb14002c8 (DEFAULT_TABLE table=test-table [id=cb72e5ad77b748509a50fe07177da74b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:12:14.394862 17564 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f1ffe4cb8bf94360892a42ebb14002c8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:12:14.406563 17915 tablet_bootstrap.cc:492] T f1ffe4cb8bf94360892a42ebb14002c8 P 327bc76b8ba447fb821d916ca281b6a5: Bootstrap starting.
I20260928 02:12:14.408406 17916 tablet_bootstrap.cc:492] T f1ffe4cb8bf94360892a42ebb14002c8 P 4d8ac0bbb79f46acaa70becb4a39edc3: Bootstrap starting.
I20260928 02:12:14.410106 17917 tablet_bootstrap.cc:492] T f1ffe4cb8bf94360892a42ebb14002c8 P add5f7c598324480835f9fdc87340fc7: Bootstrap starting.
I20260928 02:12:14.410863 17915 tablet_bootstrap.cc:654] T f1ffe4cb8bf94360892a42ebb14002c8 P 327bc76b8ba447fb821d916ca281b6a5: Neither blocks nor log segments found. Creating new log.
I20260928 02:12:14.411980 17916 tablet_bootstrap.cc:654] T f1ffe4cb8bf94360892a42ebb14002c8 P 4d8ac0bbb79f46acaa70becb4a39edc3: Neither blocks nor log segments found. Creating new log.
I20260928 02:12:14.412195 17915 log.cc:824] T f1ffe4cb8bf94360892a42ebb14002c8 P 327bc76b8ba447fb821d916ca281b6a5: Log is configured to *not* fsync() on all Append() calls
I20260928 02:12:14.413223 17917 tablet_bootstrap.cc:654] T f1ffe4cb8bf94360892a42ebb14002c8 P add5f7c598324480835f9fdc87340fc7: Neither blocks nor log segments found. Creating new log.
I20260928 02:12:14.413276 17916 log.cc:824] T f1ffe4cb8bf94360892a42ebb14002c8 P 4d8ac0bbb79f46acaa70becb4a39edc3: Log is configured to *not* fsync() on all Append() calls
I20260928 02:12:14.414486 17917 log.cc:824] T f1ffe4cb8bf94360892a42ebb14002c8 P add5f7c598324480835f9fdc87340fc7: Log is configured to *not* fsync() on all Append() calls
I20260928 02:12:14.416234 17915 tablet_bootstrap.cc:492] T f1ffe4cb8bf94360892a42ebb14002c8 P 327bc76b8ba447fb821d916ca281b6a5: No bootstrap required, opened a new log
I20260928 02:12:14.416283 17916 tablet_bootstrap.cc:492] T f1ffe4cb8bf94360892a42ebb14002c8 P 4d8ac0bbb79f46acaa70becb4a39edc3: No bootstrap required, opened a new log
I20260928 02:12:14.416577 17915 ts_tablet_manager.cc:1397] T f1ffe4cb8bf94360892a42ebb14002c8 P 327bc76b8ba447fb821d916ca281b6a5: Time spent bootstrapping tablet: real 0.010s user 0.000s sys 0.008s
I20260928 02:12:14.416577 17916 ts_tablet_manager.cc:1397] T f1ffe4cb8bf94360892a42ebb14002c8 P 4d8ac0bbb79f46acaa70becb4a39edc3: Time spent bootstrapping tablet: real 0.008s user 0.007s sys 0.000s
I20260928 02:12:14.423405 17917 tablet_bootstrap.cc:492] T f1ffe4cb8bf94360892a42ebb14002c8 P add5f7c598324480835f9fdc87340fc7: No bootstrap required, opened a new log
I20260928 02:12:14.423731 17917 ts_tablet_manager.cc:1397] T f1ffe4cb8bf94360892a42ebb14002c8 P add5f7c598324480835f9fdc87340fc7: Time spent bootstrapping tablet: real 0.014s user 0.005s sys 0.007s
I20260928 02:12:14.427717 17916 raft_consensus.cc:348] T f1ffe4cb8bf94360892a42ebb14002c8 P 4d8ac0bbb79f46acaa70becb4a39edc3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "327bc76b8ba447fb821d916ca281b6a5" member_type: VOTER last_known_addr { host: "127.6.235.65" port: 34811 } } peers { permanent_uuid: "add5f7c598324480835f9fdc87340fc7" member_type: VOTER last_known_addr { host: "127.6.235.67" port: 44873 } } peers { permanent_uuid: "4d8ac0bbb79f46acaa70becb4a39edc3" member_type: VOTER last_known_addr { host: "127.6.235.66" port: 39425 } }
I20260928 02:12:14.427709 17915 raft_consensus.cc:348] T f1ffe4cb8bf94360892a42ebb14002c8 P 327bc76b8ba447fb821d916ca281b6a5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "327bc76b8ba447fb821d916ca281b6a5" member_type: VOTER last_known_addr { host: "127.6.235.65" port: 34811 } } peers { permanent_uuid: "add5f7c598324480835f9fdc87340fc7" member_type: VOTER last_known_addr { host: "127.6.235.67" port: 44873 } } peers { permanent_uuid: "4d8ac0bbb79f46acaa70becb4a39edc3" member_type: VOTER last_known_addr { host: "127.6.235.66" port: 39425 } }
I20260928 02:12:14.428221 17915 raft_consensus.cc:374] T f1ffe4cb8bf94360892a42ebb14002c8 P 327bc76b8ba447fb821d916ca281b6a5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:12:14.428246 17916 raft_consensus.cc:374] T f1ffe4cb8bf94360892a42ebb14002c8 P 4d8ac0bbb79f46acaa70becb4a39edc3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:12:14.428359 17915 raft_consensus.cc:729] T f1ffe4cb8bf94360892a42ebb14002c8 P 327bc76b8ba447fb821d916ca281b6a5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 327bc76b8ba447fb821d916ca281b6a5, State: Initialized, Role: FOLLOWER
I20260928 02:12:14.428352 17916 raft_consensus.cc:729] T f1ffe4cb8bf94360892a42ebb14002c8 P 4d8ac0bbb79f46acaa70becb4a39edc3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4d8ac0bbb79f46acaa70becb4a39edc3, State: Initialized, Role: FOLLOWER
I20260928 02:12:14.429078 17916 consensus_queue.cc:260] T f1ffe4cb8bf94360892a42ebb14002c8 P 4d8ac0bbb79f46acaa70becb4a39edc3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "327bc76b8ba447fb821d916ca281b6a5" member_type: VOTER last_known_addr { host: "127.6.235.65" port: 34811 } } peers { permanent_uuid: "add5f7c598324480835f9fdc87340fc7" member_type: VOTER last_known_addr { host: "127.6.235.67" port: 44873 } } peers { permanent_uuid: "4d8ac0bbb79f46acaa70becb4a39edc3" member_type: VOTER last_known_addr { host: "127.6.235.66" port: 39425 } }
I20260928 02:12:14.429155 17915 consensus_queue.cc:260] T f1ffe4cb8bf94360892a42ebb14002c8 P 327bc76b8ba447fb821d916ca281b6a5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "327bc76b8ba447fb821d916ca281b6a5" member_type: VOTER last_known_addr { host: "127.6.235.65" port: 34811 } } peers { permanent_uuid: "add5f7c598324480835f9fdc87340fc7" member_type: VOTER last_known_addr { host: "127.6.235.67" port: 44873 } } peers { permanent_uuid: "4d8ac0bbb79f46acaa70becb4a39edc3" member_type: VOTER last_known_addr { host: "127.6.235.66" port: 39425 } }
I20260928 02:12:14.431399 17915 ts_tablet_manager.cc:1428] T f1ffe4cb8bf94360892a42ebb14002c8 P 327bc76b8ba447fb821d916ca281b6a5: Time spent starting tablet: real 0.015s user 0.011s sys 0.002s
I20260928 02:12:14.432597 17763 heartbeater.cc:503] Master 127.6.235.126:38277 was elected leader, sending a full tablet report...
I20260928 02:12:14.432972 17917 raft_consensus.cc:348] T f1ffe4cb8bf94360892a42ebb14002c8 P add5f7c598324480835f9fdc87340fc7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "327bc76b8ba447fb821d916ca281b6a5" member_type: VOTER last_known_addr { host: "127.6.235.65" port: 34811 } } peers { permanent_uuid: "add5f7c598324480835f9fdc87340fc7" member_type: VOTER last_known_addr { host: "127.6.235.67" port: 44873 } } peers { permanent_uuid: "4d8ac0bbb79f46acaa70becb4a39edc3" member_type: VOTER last_known_addr { host: "127.6.235.66" port: 39425 } }
I20260928 02:12:14.433374 17916 ts_tablet_manager.cc:1428] T f1ffe4cb8bf94360892a42ebb14002c8 P 4d8ac0bbb79f46acaa70becb4a39edc3: Time spent starting tablet: real 0.017s user 0.012s sys 0.005s
I20260928 02:12:14.433468 17917 raft_consensus.cc:374] T f1ffe4cb8bf94360892a42ebb14002c8 P add5f7c598324480835f9fdc87340fc7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:12:14.433565 17917 raft_consensus.cc:729] T f1ffe4cb8bf94360892a42ebb14002c8 P add5f7c598324480835f9fdc87340fc7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: add5f7c598324480835f9fdc87340fc7, State: Initialized, Role: FOLLOWER
I20260928 02:12:14.434278 17917 consensus_queue.cc:260] T f1ffe4cb8bf94360892a42ebb14002c8 P add5f7c598324480835f9fdc87340fc7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "327bc76b8ba447fb821d916ca281b6a5" member_type: VOTER last_known_addr { host: "127.6.235.65" port: 34811 } } peers { permanent_uuid: "add5f7c598324480835f9fdc87340fc7" member_type: VOTER last_known_addr { host: "127.6.235.67" port: 44873 } } peers { permanent_uuid: "4d8ac0bbb79f46acaa70becb4a39edc3" member_type: VOTER last_known_addr { host: "127.6.235.66" port: 39425 } }
I20260928 02:12:14.435452 17896 heartbeater.cc:503] Master 127.6.235.126:38277 was elected leader, sending a full tablet report...
I20260928 02:12:14.436126 17917 ts_tablet_manager.cc:1428] T f1ffe4cb8bf94360892a42ebb14002c8 P add5f7c598324480835f9fdc87340fc7: Time spent starting tablet: real 0.012s user 0.011s sys 0.002s
I20260928 02:12:14.488663 17922 raft_consensus.cc:482] T f1ffe4cb8bf94360892a42ebb14002c8 P 4d8ac0bbb79f46acaa70becb4a39edc3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 02:12:14.489069 17922 raft_consensus.cc:504] T f1ffe4cb8bf94360892a42ebb14002c8 P 4d8ac0bbb79f46acaa70becb4a39edc3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "327bc76b8ba447fb821d916ca281b6a5" member_type: VOTER last_known_addr { host: "127.6.235.65" port: 34811 } } peers { permanent_uuid: "add5f7c598324480835f9fdc87340fc7" member_type: VOTER last_known_addr { host: "127.6.235.67" port: 44873 } } peers { permanent_uuid: "4d8ac0bbb79f46acaa70becb4a39edc3" member_type: VOTER last_known_addr { host: "127.6.235.66" port: 39425 } }
W20260928 02:12:14.490446 17630 tablet.cc:2369] T f1ffe4cb8bf94360892a42ebb14002c8 P 327bc76b8ba447fb821d916ca281b6a5: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 02:12:14.491226 17922 leader_election.cc:288] T f1ffe4cb8bf94360892a42ebb14002c8 P 4d8ac0bbb79f46acaa70becb4a39edc3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 327bc76b8ba447fb821d916ca281b6a5 (127.6.235.65:34811), add5f7c598324480835f9fdc87340fc7 (127.6.235.67:44873)
I20260928 02:12:14.502017 17912 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:12:14.491748 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.235.66:44917 (local address 127.6.235.67:44873)
0928 02:12:14.491964 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0928 02:12:14.491971 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 02:12:14.492696 (+ 725us) server_negotiation.cc:407] Connection header received
0928 02:12:14.493066 (+ 370us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:12:14.493072 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:12:14.493150 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:12:14.493248 (+ 98us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:12:14.494181 (+ 933us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:12:14.495295 (+ 1114us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:12:14.499071 (+ 3776us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:12:14.500431 (+ 1360us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:12:14.500637 (+ 206us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:12:14.500716 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0928 02:12:14.501555 (+ 839us) server_negotiation.cc:299] Negotiation successful
0928 02:12:14.501700 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260928 02:12:14.502094 17910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:12:14.491776 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.235.66:55581 (local address 127.6.235.65:34811)
0928 02:12:14.491964 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0928 02:12:14.491971 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 02:12:14.492696 (+ 725us) server_negotiation.cc:407] Connection header received
0928 02:12:14.493066 (+ 370us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:12:14.493071 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:12:14.493150 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:12:14.493249 (+ 99us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:12:14.494181 (+ 932us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:12:14.495295 (+ 1114us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:12:14.499010 (+ 3715us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:12:14.500443 (+ 1433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:12:14.500642 (+ 199us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:12:14.500716 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0928 02:12:14.501555 (+ 839us) server_negotiation.cc:299] Negotiation successful
0928 02:12:14.501700 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260928 02:12:14.502345 17925 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:12:14.491436 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.235.65:34811 (local address 127.6.235.66:55581)
0928 02:12:14.492546 (+ 1110us) negotiation.cc:107] Waiting for socket to connect
0928 02:12:14.492571 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0928 02:12:14.492763 (+ 192us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:12:14.493302 (+ 539us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:12:14.493307 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:12:14.493342 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:12:14.494034 (+ 692us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:12:14.494052 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:12:14.495481 (+ 1429us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:12:14.495485 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:12:14.498727 (+ 3242us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:12:14.498748 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:12:14.500966 (+ 2218us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:12:14.500974 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:12:14.501272 (+ 298us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:12:14.501371 (+ 99us) client_negotiation.cc:769] Sending connection context
0928 02:12:14.501502 (+ 131us) client_negotiation.cc:240] Negotiation successful
0928 02:12:14.501707 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":927,"mutex_wait_us":486,"spinlock_wait_cycles":17408,"thread_start_us":809,"threads_started":1}
I20260928 02:12:14.502388 17924 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:12:14.491565 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.235.67:44873 (local address 127.6.235.66:44917)
0928 02:12:14.492567 (+ 1002us) negotiation.cc:107] Waiting for socket to connect
0928 02:12:14.492580 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 02:12:14.492763 (+ 183us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:12:14.493302 (+ 539us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:12:14.493307 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:12:14.493345 (+ 38us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:12:14.493956 (+ 611us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:12:14.493977 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:12:14.495479 (+ 1502us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:12:14.495486 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:12:14.498727 (+ 3241us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:12:14.498748 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:12:14.500964 (+ 2216us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:12:14.500974 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:12:14.501278 (+ 304us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:12:14.501370 (+ 92us) client_negotiation.cc:769] Sending connection context
0928 02:12:14.501502 (+ 132us) client_negotiation.cc:240] Negotiation successful
0928 02:12:14.501707 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":893,"mutex_wait_us":351,"spinlock_wait_cycles":33024,"thread_start_us":589,"threads_started":1}
I20260928 02:12:14.503695 17584 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f1ffe4cb8bf94360892a42ebb14002c8" candidate_uuid: "4d8ac0bbb79f46acaa70becb4a39edc3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "327bc76b8ba447fb821d916ca281b6a5" is_pre_election: true
I20260928 02:12:14.503813 17851 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f1ffe4cb8bf94360892a42ebb14002c8" candidate_uuid: "4d8ac0bbb79f46acaa70becb4a39edc3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "add5f7c598324480835f9fdc87340fc7" is_pre_election: true
I20260928 02:12:14.504406 17584 raft_consensus.cc:2454] T f1ffe4cb8bf94360892a42ebb14002c8 P 327bc76b8ba447fb821d916ca281b6a5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d8ac0bbb79f46acaa70becb4a39edc3 in term 0.
I20260928 02:12:14.504482 17851 raft_consensus.cc:2454] T f1ffe4cb8bf94360892a42ebb14002c8 P add5f7c598324480835f9fdc87340fc7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d8ac0bbb79f46acaa70becb4a39edc3 in term 0.
I20260928 02:12:14.505265 17654 leader_election.cc:302] T f1ffe4cb8bf94360892a42ebb14002c8 P 4d8ac0bbb79f46acaa70becb4a39edc3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 327bc76b8ba447fb821d916ca281b6a5, 4d8ac0bbb79f46acaa70becb4a39edc3, add5f7c598324480835f9fdc87340fc7; no voters:
I20260928 02:12:14.505769 17922 raft_consensus.cc:2781] T f1ffe4cb8bf94360892a42ebb14002c8 P 4d8ac0bbb79f46acaa70becb4a39edc3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 02:12:14.505935 17922 raft_consensus.cc:482] T f1ffe4cb8bf94360892a42ebb14002c8 P 4d8ac0bbb79f46acaa70becb4a39edc3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 02:12:14.506047 17922 raft_consensus.cc:3037] T f1ffe4cb8bf94360892a42ebb14002c8 P 4d8ac0bbb79f46acaa70becb4a39edc3 [term 0 FOLLOWER]: Advancing to term 1
W20260928 02:12:14.506624 17764 tablet.cc:2369] T f1ffe4cb8bf94360892a42ebb14002c8 P 4d8ac0bbb79f46acaa70becb4a39edc3: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 02:12:14.508849 17922 raft_consensus.cc:504] T f1ffe4cb8bf94360892a42ebb14002c8 P 4d8ac0bbb79f46acaa70becb4a39edc3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "327bc76b8ba447fb821d916ca281b6a5" member_type: VOTER last_known_addr { host: "127.6.235.65" port: 34811 } } peers { permanent_uuid: "add5f7c598324480835f9fdc87340fc7" member_type: VOTER last_known_addr { host: "127.6.235.67" port: 44873 } } peers { permanent_uuid: "4d8ac0bbb79f46acaa70becb4a39edc3" member_type: VOTER last_known_addr { host: "127.6.235.66" port: 39425 } }
I20260928 02:12:14.509567 17922 leader_election.cc:288] T f1ffe4cb8bf94360892a42ebb14002c8 P 4d8ac0bbb79f46acaa70becb4a39edc3 [CANDIDATE]: Term 1 election: Requested vote from peers 327bc76b8ba447fb821d916ca281b6a5 (127.6.235.65:34811), add5f7c598324480835f9fdc87340fc7 (127.6.235.67:44873)
I20260928 02:12:14.510169 17584 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f1ffe4cb8bf94360892a42ebb14002c8" candidate_uuid: "4d8ac0bbb79f46acaa70becb4a39edc3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "327bc76b8ba447fb821d916ca281b6a5"
I20260928 02:12:14.510321 17851 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f1ffe4cb8bf94360892a42ebb14002c8" candidate_uuid: "4d8ac0bbb79f46acaa70becb4a39edc3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "add5f7c598324480835f9fdc87340fc7"
I20260928 02:12:14.510413 17584 raft_consensus.cc:3037] T f1ffe4cb8bf94360892a42ebb14002c8 P 327bc76b8ba447fb821d916ca281b6a5 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:12:14.510578 17851 raft_consensus.cc:3037] T f1ffe4cb8bf94360892a42ebb14002c8 P add5f7c598324480835f9fdc87340fc7 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:12:14.512673 17584 raft_consensus.cc:2454] T f1ffe4cb8bf94360892a42ebb14002c8 P 327bc76b8ba447fb821d916ca281b6a5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4d8ac0bbb79f46acaa70becb4a39edc3 in term 1.
I20260928 02:12:14.513196 17654 leader_election.cc:302] T f1ffe4cb8bf94360892a42ebb14002c8 P 4d8ac0bbb79f46acaa70becb4a39edc3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 327bc76b8ba447fb821d916ca281b6a5, 4d8ac0bbb79f46acaa70becb4a39edc3; no voters:
I20260928 02:12:14.513275 17851 raft_consensus.cc:2454] T f1ffe4cb8bf94360892a42ebb14002c8 P add5f7c598324480835f9fdc87340fc7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4d8ac0bbb79f46acaa70becb4a39edc3 in term 1.
I20260928 02:12:14.513509 17922 raft_consensus.cc:2781] T f1ffe4cb8bf94360892a42ebb14002c8 P 4d8ac0bbb79f46acaa70becb4a39edc3 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:12:14.513986 17922 raft_consensus.cc:686] T f1ffe4cb8bf94360892a42ebb14002c8 P 4d8ac0bbb79f46acaa70becb4a39edc3 [term 1 LEADER]: Becoming Leader. State: Replica: 4d8ac0bbb79f46acaa70becb4a39edc3, State: Running, Role: LEADER
I20260928 02:12:14.514601 17922 consensus_queue.cc:237] T f1ffe4cb8bf94360892a42ebb14002c8 P 4d8ac0bbb79f46acaa70becb4a39edc3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "327bc76b8ba447fb821d916ca281b6a5" member_type: VOTER last_known_addr { host: "127.6.235.65" port: 34811 } } peers { permanent_uuid: "add5f7c598324480835f9fdc87340fc7" member_type: VOTER last_known_addr { host: "127.6.235.67" port: 44873 } } peers { permanent_uuid: "4d8ac0bbb79f46acaa70becb4a39edc3" member_type: VOTER last_known_addr { host: "127.6.235.66" port: 39425 } }
I20260928 02:12:14.519775 17440 catalog_manager.cc:5495] T f1ffe4cb8bf94360892a42ebb14002c8 P 4d8ac0bbb79f46acaa70becb4a39edc3 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4d8ac0bbb79f46acaa70becb4a39edc3 (127.6.235.66). New cstate: current_term: 1 leader_uuid: "4d8ac0bbb79f46acaa70becb4a39edc3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "327bc76b8ba447fb821d916ca281b6a5" member_type: VOTER last_known_addr { host: "127.6.235.65" port: 34811 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "add5f7c598324480835f9fdc87340fc7" member_type: VOTER last_known_addr { host: "127.6.235.67" port: 44873 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4d8ac0bbb79f46acaa70becb4a39edc3" member_type: VOTER last_known_addr { host: "127.6.235.66" port: 39425 } health_report { overall_health: HEALTHY } } }
W20260928 02:12:14.543958 17897 tablet.cc:2369] T f1ffe4cb8bf94360892a42ebb14002c8 P add5f7c598324480835f9fdc87340fc7: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 02:12:14.570719 17913 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:12:14.564779 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59924 (local address 127.6.235.66:39425)
0928 02:12:14.565047 (+ 268us) server_negotiation.cc:206] Beginning negotiation
0928 02:12:14.565053 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 02:12:14.565074 (+ 21us) server_negotiation.cc:407] Connection header received
0928 02:12:14.565208 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:12:14.565213 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:12:14.565323 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:12:14.565487 (+ 164us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 02:12:14.565818 (+ 331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:12:14.566900 (+ 1082us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:12:14.568180 (+ 1280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:12:14.568811 (+ 631us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:12:14.568963 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:12:14.569440 (+ 477us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 02:12:14.570152 (+ 712us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 02:12:14.570290 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0928 02:12:14.570366 (+ 76us) server_negotiation.cc:299] Negotiation successful
0928 02:12:14.570452 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":123}
I20260928 02:12:14.579530 17851 raft_consensus.cc:1261] T f1ffe4cb8bf94360892a42ebb14002c8 P add5f7c598324480835f9fdc87340fc7 [term 1 FOLLOWER]: Refusing update from remote peer 4d8ac0bbb79f46acaa70becb4a39edc3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 02:12:14.579530 17584 raft_consensus.cc:1261] T f1ffe4cb8bf94360892a42ebb14002c8 P 327bc76b8ba447fb821d916ca281b6a5 [term 1 FOLLOWER]: Refusing update from remote peer 4d8ac0bbb79f46acaa70becb4a39edc3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 02:12:14.580597 17926 consensus_queue.cc:1035] T f1ffe4cb8bf94360892a42ebb14002c8 P 4d8ac0bbb79f46acaa70becb4a39edc3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "add5f7c598324480835f9fdc87340fc7" member_type: VOTER last_known_addr { host: "127.6.235.67" port: 44873 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 02:12:14.580974 17922 consensus_queue.cc:1035] T f1ffe4cb8bf94360892a42ebb14002c8 P 4d8ac0bbb79f46acaa70becb4a39edc3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "327bc76b8ba447fb821d916ca281b6a5" member_type: VOTER last_known_addr { host: "127.6.235.65" port: 34811 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 02:12:14.592799 17930 mvcc.cc:205] Tried to move back new op lower bound from 7334140045626482688 to 7334140045376049152. Current Snapshot: MvccSnapshot[applied={T|T < 7334140045626482688}]
I20260928 02:12:14.607352 17439 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35640:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 02:12:14.607628 17439 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35640:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 02:12:14.611550 17439 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 31b79a4bef944137ba8e2a00893ad446: Sending DeleteTablet for 3 replicas of tablet f1ffe4cb8bf94360892a42ebb14002c8
I20260928 02:12:14.612946 17564 tablet_service.cc:1514] Processing DeleteTablet for tablet f1ffe4cb8bf94360892a42ebb14002c8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:12:14 UTC) from {username='kudu', principal='kudu/127.6.235.126@KRBTEST.COM'} at 127.0.0.1:45762
I20260928 02:12:14.612963 17698 tablet_service.cc:1514] Processing DeleteTablet for tablet f1ffe4cb8bf94360892a42ebb14002c8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:12:14 UTC) from {username='kudu', principal='kudu/127.6.235.126@KRBTEST.COM'} at 127.0.0.1:59922
I20260928 02:12:14.613147 17831 tablet_service.cc:1514] Processing DeleteTablet for tablet f1ffe4cb8bf94360892a42ebb14002c8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:12:14 UTC) from {username='kudu', principal='kudu/127.6.235.126@KRBTEST.COM'} at 127.0.0.1:38712
I20260928 02:12:14.616546 7085 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskTlaUuc/build/asan/bin/kudu pid 17501
I20260928 02:12:14.628587 17938 tablet_replica.cc:330] T f1ffe4cb8bf94360892a42ebb14002c8 P 327bc76b8ba447fb821d916ca281b6a5: stopping tablet replica
I20260928 02:12:14.628767 17940 tablet_replica.cc:330] T f1ffe4cb8bf94360892a42ebb14002c8 P add5f7c598324480835f9fdc87340fc7: stopping tablet replica
I20260928 02:12:14.629372 17940 raft_consensus.cc:2229] T f1ffe4cb8bf94360892a42ebb14002c8 P add5f7c598324480835f9fdc87340fc7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 02:12:14.629997 17940 raft_consensus.cc:2258] T f1ffe4cb8bf94360892a42ebb14002c8 P add5f7c598324480835f9fdc87340fc7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 02:12:14.630928 17939 tablet_replica.cc:330] T f1ffe4cb8bf94360892a42ebb14002c8 P 4d8ac0bbb79f46acaa70becb4a39edc3: stopping tablet replica
I20260928 02:12:14.631390 17939 raft_consensus.cc:2229] T f1ffe4cb8bf94360892a42ebb14002c8 P 4d8ac0bbb79f46acaa70becb4a39edc3 [term 1 LEADER]: Raft consensus shutting down.
I20260928 02:12:14.632134 17939 raft_consensus.cc:2258] T f1ffe4cb8bf94360892a42ebb14002c8 P 4d8ac0bbb79f46acaa70becb4a39edc3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 02:12:14.632381 17938 raft_consensus.cc:2229] T f1ffe4cb8bf94360892a42ebb14002c8 P 327bc76b8ba447fb821d916ca281b6a5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 02:12:14.632994 17938 raft_consensus.cc:2258] T f1ffe4cb8bf94360892a42ebb14002c8 P 327bc76b8ba447fb821d916ca281b6a5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 02:12:14.634757 17940 ts_tablet_manager.cc:1905] T f1ffe4cb8bf94360892a42ebb14002c8 P add5f7c598324480835f9fdc87340fc7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 02:12:14.635185 17938 ts_tablet_manager.cc:1905] T f1ffe4cb8bf94360892a42ebb14002c8 P 327bc76b8ba447fb821d916ca281b6a5: Deleting tablet data with delete state TABLET_DATA_DELETED
Sep 28 02:12:14 dist-test-slave-86gv krb5kdc[17395](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561532, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.235.65@KRBTEST.COM
I20260928 02:12:14.638753 17939 ts_tablet_manager.cc:1905] T f1ffe4cb8bf94360892a42ebb14002c8 P 4d8ac0bbb79f46acaa70becb4a39edc3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 02:12:14.642742 17938 ts_tablet_manager.cc:1918] T f1ffe4cb8bf94360892a42ebb14002c8 P 327bc76b8ba447fb821d916ca281b6a5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 02:12:14.643005 17938 log.cc:1196] T f1ffe4cb8bf94360892a42ebb14002c8 P 327bc76b8ba447fb821d916ca281b6a5: Deleting WAL directory at /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790561432251112-7085-0/minicluster-data/ts-0/wal/wals/f1ffe4cb8bf94360892a42ebb14002c8
I20260928 02:12:14.643638 17938 ts_tablet_manager.cc:1939] T f1ffe4cb8bf94360892a42ebb14002c8 P 327bc76b8ba447fb821d916ca281b6a5: Deleting consensus metadata
I20260928 02:12:14.644402 17940 ts_tablet_manager.cc:1918] T f1ffe4cb8bf94360892a42ebb14002c8 P add5f7c598324480835f9fdc87340fc7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 02:12:14.644582 17940 log.cc:1196] T f1ffe4cb8bf94360892a42ebb14002c8 P add5f7c598324480835f9fdc87340fc7: Deleting WAL directory at /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790561432251112-7085-0/minicluster-data/ts-2/wal/wals/f1ffe4cb8bf94360892a42ebb14002c8
I20260928 02:12:14.645942 17427 catalog_manager.cc:4841] TS 327bc76b8ba447fb821d916ca281b6a5 (127.6.235.65:34811): tablet f1ffe4cb8bf94360892a42ebb14002c8 (table test-table [id=cb72e5ad77b748509a50fe07177da74b]) successfully deleted
I20260928 02:12:14.651784 17939 ts_tablet_manager.cc:1918] T f1ffe4cb8bf94360892a42ebb14002c8 P 4d8ac0bbb79f46acaa70becb4a39edc3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 02:12:14.652467 17910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:12:14.618169 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45778 (local address 127.6.235.65:34811)
0928 02:12:14.618350 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0928 02:12:14.618356 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 02:12:14.622709 (+ 4353us) server_negotiation.cc:407] Connection header received
0928 02:12:14.623770 (+ 1061us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:12:14.623779 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:12:14.623905 (+ 126us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:12:14.624022 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:12:14.628977 (+ 4955us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:12:14.630070 (+ 1093us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:12:14.631999 (+ 1929us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:12:14.632751 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:12:14.632973 (+ 222us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:12:14.638497 (+ 5524us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:12:14.638515 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:12:14.638534 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:12:14.638585 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:12:14.642814 (+ 4229us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:12:14.648710 (+ 5896us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:12:14.648746 (+ 36us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:12:14.648750 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:12:14.649054 (+ 304us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:12:14.649880 (+ 826us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:12:14.649889 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:12:14.649893 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:12:14.650588 (+ 695us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:12:14.651044 (+ 456us) server_negotiation.cc:1035] Waiting for connection context
0928 02:12:14.651777 (+ 733us) server_negotiation.cc:299] Negotiation successful
0928 02:12:14.652034 (+ 257us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260928 02:12:14.652603 17940 ts_tablet_manager.cc:1939] T f1ffe4cb8bf94360892a42ebb14002c8 P add5f7c598324480835f9fdc87340fc7: Deleting consensus metadata
I20260928 02:12:14.653225 17624 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 02:12:14.654351 17939 log.cc:1196] T f1ffe4cb8bf94360892a42ebb14002c8 P 4d8ac0bbb79f46acaa70becb4a39edc3: Deleting WAL directory at /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790561432251112-7085-0/minicluster-data/ts-1/wal/wals/f1ffe4cb8bf94360892a42ebb14002c8
I20260928 02:12:14.654435 17426 catalog_manager.cc:4841] TS add5f7c598324480835f9fdc87340fc7 (127.6.235.67:44873): tablet f1ffe4cb8bf94360892a42ebb14002c8 (table test-table [id=cb72e5ad77b748509a50fe07177da74b]) successfully deleted
I20260928 02:12:14.655011 17939 ts_tablet_manager.cc:1939] T f1ffe4cb8bf94360892a42ebb14002c8 P 4d8ac0bbb79f46acaa70becb4a39edc3: Deleting consensus metadata
I20260928 02:12:14.656450 17426 catalog_manager.cc:4841] TS 4d8ac0bbb79f46acaa70becb4a39edc3 (127.6.235.66:39425): tablet f1ffe4cb8bf94360892a42ebb14002c8 (table test-table [id=cb72e5ad77b748509a50fe07177da74b]) successfully deleted
I20260928 02:12:14.747778 7085 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTlaUuc/build/asan/bin/kudu with pid 17501
I20260928 02:12:14.770684 7085 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskTlaUuc/build/asan/bin/kudu pid 17635
Sep 28 02:12:14 dist-test-slave-86gv krb5kdc[17395](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561532, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.235.66@KRBTEST.COM
I20260928 02:12:14.803117 17913 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:12:14.771592 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59936 (local address 127.6.235.66:39425)
0928 02:12:14.771783 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0928 02:12:14.771791 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 02:12:14.771934 (+ 143us) server_negotiation.cc:407] Connection header received
0928 02:12:14.772121 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:12:14.772127 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:12:14.772264 (+ 137us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:12:14.772386 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:12:14.773943 (+ 1557us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:12:14.775229 (+ 1286us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:12:14.777094 (+ 1865us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:12:14.777800 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:12:14.777938 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:12:14.795431 (+ 17493us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:12:14.795469 (+ 38us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:12:14.795492 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:12:14.795584 (+ 92us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:12:14.799887 (+ 4303us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:12:14.800663 (+ 776us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:12:14.800673 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:12:14.800677 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:12:14.800849 (+ 172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:12:14.801289 (+ 440us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:12:14.801299 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:12:14.801302 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:12:14.801867 (+ 565us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:12:14.802169 (+ 302us) server_negotiation.cc:1035] Waiting for connection context
0928 02:12:14.802411 (+ 242us) server_negotiation.cc:299] Negotiation successful
0928 02:12:14.802679 (+ 268us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260928 02:12:14.803994 17758 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 02:12:14.920292 7085 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTlaUuc/build/asan/bin/kudu with pid 17635
I20260928 02:12:14.940474 7085 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskTlaUuc/build/asan/bin/kudu pid 17768
Sep 28 02:12:14 dist-test-slave-86gv krb5kdc[17395](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561532, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.235.67@KRBTEST.COM
I20260928 02:12:14.957676 17912 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:12:14.941249 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38714 (local address 127.6.235.67:44873)
0928 02:12:14.941428 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0928 02:12:14.941432 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:12:14.941451 (+ 19us) server_negotiation.cc:407] Connection header received
0928 02:12:14.941595 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:12:14.941598 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:12:14.941671 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:12:14.941763 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:12:14.943391 (+ 1628us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:12:14.944223 (+ 832us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:12:14.945852 (+ 1629us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:12:14.946244 (+ 392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:12:14.946347 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:12:14.951388 (+ 5041us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:12:14.951406 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:12:14.951423 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:12:14.951469 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:12:14.954601 (+ 3132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:12:14.955363 (+ 762us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:12:14.955372 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:12:14.955374 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:12:14.955537 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:12:14.955912 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:12:14.955916 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:12:14.955918 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:12:14.956310 (+ 392us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:12:14.956589 (+ 279us) server_negotiation.cc:1035] Waiting for connection context
0928 02:12:14.957141 (+ 552us) server_negotiation.cc:299] Negotiation successful
0928 02:12:14.957416 (+ 275us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260928 02:12:14.958429 17891 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 02:12:15.052083 7085 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTlaUuc/build/asan/bin/kudu with pid 17768
I20260928 02:12:15.069814 7085 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskTlaUuc/build/asan/bin/kudu pid 17408
I20260928 02:12:15.071177 17470 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 02:12:15.179363 7085 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTlaUuc/build/asan/bin/kudu with pid 17408
2026-09-28T02:12:15Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationSecureCluster (4423 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegration
Loading random data
Initializing database '/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790561432251112-7085-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 02:12:15 dist-test-slave-86gv krb5kdc[17948](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 02:12:15 dist-test-slave-86gv krb5kdc[17948](info): set up 2 sockets
Sep 28 02:12:15 dist-test-slave-86gv krb5kdc[17948](info): commencing operation
krb5kdc: starting...
W20260928 02:12:16.315094 7085 mini_kdc.cc:121] Time spent starting KDC: real 1.091s user 0.004s sys 0.023s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790561432251112-7085-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790561432251112-7085-0/krb5kdc/test-user.keytab.
Sep 28 02:12:16 dist-test-slave-86gv krb5kdc[17948](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561536, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T02:12:16Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T02:12:16Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.6.235.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.6.235.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.6.235.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790561432251112-7085-0/krb5kdc/rangeradmin_127.6.235.84@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.6.235.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790561432251112-7085-0/krb5kdc/rangeradmin_127.6.235.84@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.6.235.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.6.235.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.6.235.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790561432251112-7085-0/krb5kdc/rangerlookup_127.6.235.84@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.6.235.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790561432251112-7085-0/krb5kdc/rangerlookup_127.6.235.84@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.6.235.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.6.235.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.6.235.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790561432251112-7085-0/krb5kdc/HTTP_127.6.235.84@KRBTEST.COM.keytab.
Entry for principal HTTP/127.6.235.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790561432251112-7085-0/krb5kdc/HTTP_127.6.235.84@KRBTEST.COM.keytab.
I20260928 02:12:16.676517 7085 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790561432251112-7085-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskTlaUuc/build/asan/bin/postgres/pg_ctl -D /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790561432251112-7085-0/minicluster-data/postgres -l logfile start
2026-09-28 02:12:19.441 UTC [17978] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-28 02:12:19.441 UTC [17978] LOG: listening on IPv4 address "127.6.235.84", port 54087
2026-09-28 02:12:19.445 UTC [17978] LOG: listening on Unix socket "/tmp/.s.PGSQL.54087"
2026-09-28 02:12:19.468 UTC [17981] LOG: database system was shut down at 2026-09-28 02:12:17 UTC
2026-09-28 02:12:19.472 UTC [17978] LOG: database system is ready to accept connections
I20260928 02:12:20.466328 7085 mini_postgres.cc:96] Postgres bound to 54087
2026-09-28 02:12:20.485 UTC [17989] FATAL: database "slave" does not exist
127.6.235.84:54087 - accepting connections
I20260928 02:12:20.486404 7085 mini_ranger.cc:161] Starting Ranger...
I20260928 02:12:20.515058 7085 mini_ranger.cc:84] Created miniranger Postgres user
I20260928 02:12:21.091114 7085 mini_ranger.cc:87] Created ranger Postgres database
I20260928 02:12:21.091297 7085 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790561432251112-7085-0/minicluster-data/ranger-admin
2026-09-28 02:12:21,389 [I] DB FLAVOR :POSTGRES
2026-09-28 02:12:21,389 [I] --------- Verifying Ranger DB connection ---------
2026-09-28 02:12:21,389 [I] Checking connection..
2026-09-28 02:12:21,390 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskTlaUuc/build/asan/bin/postgresql.jar:/tmp/dist-test-taskTlaUuc/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.235.84:54087/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-28 02:12:21,686 [I] Checking connection passed.
2026-09-28 02:12:21,686 [I] --------- Verifying version history table ---------
2026-09-28 02:12:21,686 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskTlaUuc/build/asan/bin/postgresql.jar:/tmp/dist-test-taskTlaUuc/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.235.84:54087/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-28 02:12:21,930 [I] Table x_db_version_h does not exist in database ranger
2026-09-28 02:12:21,931 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskTlaUuc/build/asan/bin/postgresql.jar:/tmp/dist-test-taskTlaUuc/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.235.84:54087/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-28 02:12:22,185 [I] Table x_db_version_h does not exist in database ranger
2026-09-28 02:12:22,185 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-28 02:12:22,185 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskTlaUuc/build/asan/bin/postgresql.jar:/tmp/dist-test-taskTlaUuc/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.235.84:54087/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskTlaUuc/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-28 02:12:22.434 UTC [18053] WARNING: there is no transaction in progress
2026-09-28 02:12:22,441 [I] create_dbversion_catalog.sql file imported successfully
2026-09-28 02:12:22,441 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskTlaUuc/build/asan/bin/postgresql.jar:/tmp/dist-test-taskTlaUuc/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.235.84:54087/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-28 02:12:22,681 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-28 02:12:22,682 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-28 02:12:22,682 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskTlaUuc/build/asan/bin/postgresql.jar:/tmp/dist-test-taskTlaUuc/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.235.84:54087/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-28 02:12:22,937 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskTlaUuc/build/asan/bin/postgresql.jar:/tmp/dist-test-taskTlaUuc/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.235.84:54087/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-28 02:12:23,200 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskTlaUuc/build/asan/bin/postgresql.jar:/tmp/dist-test-taskTlaUuc/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.235.84:54087/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-86gv.c.gcp-upstream.internal','N') ;"
2026-09-28 02:12:23,436 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskTlaUuc/build/asan/bin/postgresql.jar:/tmp/dist-test-taskTlaUuc/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.235.84:54087/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-28 02:12:23,695 [I] Table x_portal_user does not exist in database ranger
2026-09-28 02:12:23,696 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskTlaUuc/build/asan/bin/postgresql.jar:/tmp/dist-test-taskTlaUuc/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.235.84:54087/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-28 02:12:23,961 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-28 02:12:23,961 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskTlaUuc/build/asan/bin/postgresql.jar:/tmp/dist-test-taskTlaUuc/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.235.84:54087/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-28 02:12:24,200 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskTlaUuc/build/asan/bin/postgresql.jar:/tmp/dist-test-taskTlaUuc/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.235.84:54087/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-28 02:12:24,433 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-28 02:12:24,434 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskTlaUuc/build/asan/bin/postgresql.jar:/tmp/dist-test-taskTlaUuc/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.235.84:54087/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskTlaUuc/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-28 02:12:25.447 UTC [18161] WARNING: there is no transaction in progress
2026-09-28 02:12:25.467 UTC [18161] WARNING: there is no transaction in progress
2026-09-28 02:12:25.485 UTC [18161] WARNING: there is no transaction in progress
2026-09-28 02:12:25.502 UTC [18161] WARNING: there is no transaction in progress
2026-09-28 02:12:25.521 UTC [18161] WARNING: there is no transaction in progress
2026-09-28 02:12:25.542 UTC [18161] WARNING: there is no transaction in progress
2026-09-28 02:12:25.609 UTC [18161] WARNING: there is no transaction in progress
2026-09-28 02:12:25.617 UTC [18161] WARNING: there is no transaction in progress
2026-09-28 02:12:25.631 UTC [18161] WARNING: there is no transaction in progress
2026-09-28 02:12:25.643 UTC [18161] WARNING: there is no transaction in progress
2026-09-28 02:12:25.653 UTC [18161] WARNING: there is no transaction in progress
2026-09-28 02:12:25.669 UTC [18161] WARNING: there is no transaction in progress
2026-09-28 02:12:25.680 UTC [18161] WARNING: there is no transaction in progress
2026-09-28 02:12:25.687 UTC [18161] WARNING: there is no transaction in progress
2026-09-28 02:12:25.694 UTC [18161] WARNING: there is no transaction in progress
2026-09-28 02:12:25.701 UTC [18161] WARNING: there is no transaction in progress
2026-09-28 02:12:26,391 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-28 02:12:26,391 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskTlaUuc/build/asan/bin/postgresql.jar:/tmp/dist-test-taskTlaUuc/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.235.84:54087/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-28 02:12:26,653 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-28 02:12:26,654 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskTlaUuc/build/asan/bin/postgresql.jar:/tmp/dist-test-taskTlaUuc/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.235.84:54087/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-28 02:12:26,899 [I] Table x_portal_user already exists in database 'ranger'
2026-09-28 02:12:26,900 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskTlaUuc/build/asan/bin/postgresql.jar:/tmp/dist-test-taskTlaUuc/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.235.84:54087/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-28 02:12:27,165 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-28 02:12:27,166 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskTlaUuc/build/asan/bin/postgresql.jar:/tmp/dist-test-taskTlaUuc/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.235.84:54087/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-28 02:12:27,409 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskTlaUuc/build/asan/bin/postgresql.jar:/tmp/dist-test-taskTlaUuc/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.235.84:54087/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-28 02:12:27,685 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskTlaUuc/build/asan/bin/postgresql.jar:/tmp/dist-test-taskTlaUuc/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.235.84:54087/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-86gv.c.gcp-upstream.internal';"
2026-09-28 02:12:27,913 [I] CORE_DB_SCHEMA import status has been updated
2026-09-28 02:12:27,914 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskTlaUuc/build/asan/bin/postgresql.jar:/tmp/dist-test-taskTlaUuc/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.235.84:54087/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-28 02:12:28,190 [I] DB_PATCHES have already been applied
I20260928 02:12:28.196776 7085 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790561432251112-7085-0/minicluster-data/ranger-admin:/tmp/dist-test-taskTlaUuc/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskTlaUuc/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskTlaUuc/build/asan/bin/postgresql.jar:/tmp/dist-test-taskTlaUuc/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260928 02:12:28.196957 7085 mini_ranger.cc:193] Using host: 127.6.235.84
I20260928 02:12:28.213102 7085 mini_ranger.cc:227] Ranger admin URL: http://127.6.235.84:35643
Sep 28, 2026 2:12:28 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 28, 2026 2:12:28 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 28, 2026 2:12:28 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790561432251112-7085-0/minicluster-data/ranger-admin/ews/webapp
Sep 28, 2026 2:12:28 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790561432251112-7085-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 28, 2026 2:12:28 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790561432251112-7085-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 28, 2026 2:12:29 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790561432251112-7085-0/minicluster-data/ranger-admin/ews/webapp].
Sep 28, 2026 2:12:29 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260928 02:12:29.375481 7085 mini_ranger.cc:160] Time spent starting Ranger: real 8.889s user 0.002s sys 0.043s
Sep 28, 2026 2:12:49 AM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 28, 2026 2:12:49 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.UserREST
Sep 28, 2026 2:12:49 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 28, 2026 2:12:49 AM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 28, 2026 2:12:49 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 28, 2026 2:12:49 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 28, 2026 2:12:49 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 28, 2026 2:12:49 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 28, 2026 2:12:49 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 28, 2026 2:12:49 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 28, 2026 2:12:49 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 28, 2026 2:12:49 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 28, 2026 2:12:49 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 28, 2026 2:12:49 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 28, 2026 2:12:49 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 28, 2026 2:12:49 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 28, 2026 2:12:49 AM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 28, 2026 2:12:50 AM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260928 02:12:51.018857 7085 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.6.235.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.6.235.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.6.235.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790561432251112-7085-0/krb5kdc/rangerkms_127.6.235.84@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.6.235.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790561432251112-7085-0/krb5kdc/rangerkms_127.6.235.84@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.6.235.84@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.6.235.84@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.6.235.84@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790561432251112-7085-0/krb5kdc/HTTP_127.6.235.84@KRBTEST.COM.keytab.
Entry for principal HTTP/127.6.235.84@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790561432251112-7085-0/krb5kdc/HTTP_127.6.235.84@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 28 02:12:51 dist-test-slave-86gv krb5kdc[17948](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561571, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260928 02:12:51.207837 7085 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260928 02:12:51.231170 7085 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260928 02:12:52.236158 7085 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260928 02:12:52.236300 7085 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790561432251112-7085-0/minicluster-data/ranger-kms
I20260928 02:12:52.236449 7085 mini_ranger_kms.cc:227] Using postgres at 127.6.235.84:54087
2026-09-28 02:12:52,682 [I] DB FLAVOR :POSTGRES
2026-09-28 02:12:52,682 [I] --------- Verifying Ranger DB connection ---------
2026-09-28 02:12:52,682 [I] Checking connection
2026-09-28 02:12:52,682 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskTlaUuc/build/asan/bin/postgresql.jar:/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790561432251112-7085-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.235.84:54087/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-28 02:12:52,949 [I] connection success
2026-09-28 02:12:52,949 [I] --------- Verifying Ranger DB tables ---------
2026-09-28 02:12:52,949 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-28 02:12:52,949 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskTlaUuc/build/asan/bin/postgresql.jar:/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790561432251112-7085-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.235.84:54087/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-28 02:12:53,201 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-28 02:12:53,201 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-28 02:12:53,201 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-28 02:12:53,201 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskTlaUuc/build/asan/bin/postgresql.jar:/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790561432251112-7085-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.235.84:54087/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790561432251112-7085-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-28 02:12:53,473 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260928 02:12:53.963841 7085 mini_ranger_kms.cc:310] Created kms service
I20260928 02:12:54.221542 7085 mini_ranger_kms.cc:326] Created kudu user
I20260928 02:12:54.266252 7085 mini_ranger_kms.cc:343] Created rangerkms user
Sep 28 02:12:54 dist-test-slave-86gv krb5kdc[17948](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561571, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.6.235.84@KRBTEST.COM
I20260928 02:12:54.638206 7085 mini_ranger_kms.cc:384] Added ranger policy
I20260928 02:12:54.638520 7085 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790561432251112-7085-0/minicluster-data/ranger-kms:/tmp/dist-test-taskTlaUuc/build/asan/bin/postgresql.jar:/tmp/dist-test-taskTlaUuc/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskTlaUuc/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260928 02:12:54.638607 7085 mini_ranger_kms.cc:240] Using host: 127.6.235.84
I20260928 02:12:54.665077 7085 mini_ranger_kms.cc:276] Ranger KMS PID: 18341
I20260928 02:12:54.665277 7085 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.6.235.84:42099
Sep 28, 2026 2:12:55 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 28, 2026 2:12:55 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 28, 2026 2:12:56 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790561432251112-7085-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 28, 2026 2:12:56 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790561432251112-7085-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 28, 2026 2:12:56 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790561432251112-7085-0/minicluster-data/ranger-kms/ews/webapp].
Sep 28, 2026 2:12:56 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260928 02:12:57.044550 7085 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.837s user 0.153s sys 0.307s
I20260928 02:12:57.044804 7085 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260928 02:12:57.045058 7085 mini_ranger_kms.cc:401] 127.6.235.84:42099/kms/v1/keys
Sep 28 02:12:57 dist-test-slave-86gv krb5kdc[17948](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790561577, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.6.235.84@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Sep 28 02:12:57 dist-test-slave-86gv krb5kdc[17948](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561577, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.6.235.84@KRBTEST.COM for HTTP/127.6.235.84@KRBTEST.COM
Sep 28 02:12:58 dist-test-slave-86gv krb5kdc[17948](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561577, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.6.235.84@KRBTEST.COM for HTTP/127.6.235.84@KRBTEST.COM
Sep 28 02:13:00 dist-test-slave-86gv krb5kdc[17948](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561580, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.6.235.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.235.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.235.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790561432251112-7085-0/krb5kdc/kudu_127.6.235.126.keytab.
Entry for principal kudu/127.6.235.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790561432251112-7085-0/krb5kdc/kudu_127.6.235.126.keytab.
I20260928 02:13:00.393790 7085 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTlaUuc/build/asan/bin/kudu
/tmp/dist-test-taskTlaUuc/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790561432251112-7085-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790561432251112-7085-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790561432251112-7085-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790561432251112-7085-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.235.126:36441
--webserver_interface=127.6.235.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790561432251112-7085-0/krb5kdc/kudu_127.6.235.126.keytab
--principal=kudu/127.6.235.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.235.84:39613
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.235.126:36441
--ranger_config_path=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790561432251112-7085-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.6.235.84:42099/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790561432251112-7085-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790561432251112-7085-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790561432251112-7085-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:13:00.732923 18392 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:13:00.733229 18392 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:13:00.733392 18392 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:13:00.740852 18392 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 02:13:00.741027 18392 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:13:00.741130 18392 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:13:00.741218 18392 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 02:13:00.741302 18392 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 02:13:00.751650 18392 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.235.84:39613
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790561432251112-7085-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790561432251112-7085-0/minicluster-data/master-0/wal
--ranger_kms_url=127.6.235.84:42099/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.6.235.126:36441
--ranger_config_path=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790561432251112-7085-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790561432251112-7085-0/krb5kdc/kudu_127.6.235.126.keytab
--principal=kudu/127.6.235.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.235.126:36441
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790561432251112-7085-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.235.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790561432251112-7085-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 00:51:59 UTC on ccycloud.quasar-axgkwn.root.comops.site
build id 4676464
ASAN enabled
I20260928 02:13:00.753121 18392 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:13:00.754423 18392 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:13:00.766672 18397 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:13:00.766654 18400 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:13:00.766793 18398 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:13:00.767495 18392 server_base.cc:1034] running on GCE node
I20260928 02:13:00.768268 18392 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:13:00.769804 18392 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:13:00.771032 18392 hybrid_clock.cc:648] HybridClock initialized: now 1790561580770965 us; error 59 us; skew 500 ppm
I20260928 02:13:00.771381 18392 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 02:13:00 dist-test-slave-86gv krb5kdc[17948](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561580, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.235.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:13:00.774644 18392 init.cc:377] Logged in from keytab as kudu/127.6.235.126@KRBTEST.COM (short username kudu)
I20260928 02:13:00.776156 18392 webserver.cc:460] Webserver started at http://127.6.235.126:32981/ using document root <none> and password file <none>
I20260928 02:13:00.776800 18392 fs_manager.cc:359] Metadata directory not provided
I20260928 02:13:00.776932 18392 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:13:00.777256 18392 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 28 02:13:00 dist-test-slave-86gv krb5kdc[17948](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561580, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.235.126@KRBTEST.COM for HTTP/127.6.235.84@KRBTEST.COM
I20260928 02:13:00.845098 18392 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790561432251112-7085-0/minicluster-data/master-0/data/instance:
uuid: "e09f2d99b867436286d683dc57e441fa"
format_stamp: "Formatted at 2026-09-28 02:13:00 on dist-test-slave-86gv"
server_key: "ca07abee10d87fdebf812be5cfa5d657"
server_key_iv: "99894f53165729cc8af0bb72f4baa3f5"
server_key_version: "kuduclusterkey@0"
I20260928 02:13:00.846306 18392 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790561432251112-7085-0/minicluster-data/master-0/wal/instance:
uuid: "e09f2d99b867436286d683dc57e441fa"
format_stamp: "Formatted at 2026-09-28 02:13:00 on dist-test-slave-86gv"
server_key: "ca07abee10d87fdebf812be5cfa5d657"
server_key_iv: "99894f53165729cc8af0bb72f4baa3f5"
server_key_version: "kuduclusterkey@0"
I20260928 02:13:00.853686 18392 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.002s sys 0.005s
I20260928 02:13:00.874773 18407 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:13:00.876447 18392 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.000s sys 0.004s
I20260928 02:13:00.876645 18392 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790561432251112-7085-0/minicluster-data/master-0/data,/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790561432251112-7085-0/minicluster-data/master-0/wal
uuid: "e09f2d99b867436286d683dc57e441fa"
format_stamp: "Formatted at 2026-09-28 02:13:00 on dist-test-slave-86gv"
server_key: "ca07abee10d87fdebf812be5cfa5d657"
server_key_iv: "99894f53165729cc8af0bb72f4baa3f5"
server_key_version: "kuduclusterkey@0"
I20260928 02:13:00.876883 18392 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790561432251112-7085-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790561432251112-7085-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790561432251112-7085-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:13:00.901738 18392 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:13:00.906038 18392 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:13:00.906371 18392 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:13:00.926815 18392 rpc_server.cc:307] RPC server started. Bound to: 127.6.235.126:36441
I20260928 02:13:00.926836 18458 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.235.126:36441 every 8 connection(s)
I20260928 02:13:00.928349 18392 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790561432251112-7085-0/minicluster-data/master-0/data/info.pb
I20260928 02:13:00.932365 7085 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTlaUuc/build/asan/bin/kudu as pid 18392
I20260928 02:13:00.932574 7085 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790561432251112-7085-0/minicluster-data/master-0/wal/instance
I20260928 02:13:00.932626 18459 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:13:00.933382 7085 external_mini_cluster.cc:1428] Setting key e02d81c43af255f495ab01cfe58ffc7d
I20260928 02:13:00.950743 18459 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e09f2d99b867436286d683dc57e441fa: Bootstrap starting.
I20260928 02:13:00.954725 18459 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e09f2d99b867436286d683dc57e441fa: Neither blocks nor log segments found. Creating new log.
I20260928 02:13:00.956049 18459 log.cc:824] T 00000000000000000000000000000000 P e09f2d99b867436286d683dc57e441fa: Log is configured to *not* fsync() on all Append() calls
I20260928 02:13:00.959141 18459 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e09f2d99b867436286d683dc57e441fa: No bootstrap required, opened a new log
Sep 28 02:13:00 dist-test-slave-86gv krb5kdc[17948](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561580, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.235.126@KRBTEST.COM
I20260928 02:13:00.968549 18459 raft_consensus.cc:348] T 00000000000000000000000000000000 P e09f2d99b867436286d683dc57e441fa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e09f2d99b867436286d683dc57e441fa" member_type: VOTER last_known_addr { host: "127.6.235.126" port: 36441 } }
I20260928 02:13:00.969123 18459 raft_consensus.cc:374] T 00000000000000000000000000000000 P e09f2d99b867436286d683dc57e441fa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:13:00.969242 18459 raft_consensus.cc:729] T 00000000000000000000000000000000 P e09f2d99b867436286d683dc57e441fa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e09f2d99b867436286d683dc57e441fa, State: Initialized, Role: FOLLOWER
I20260928 02:13:00.969997 18459 consensus_queue.cc:260] T 00000000000000000000000000000000 P e09f2d99b867436286d683dc57e441fa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e09f2d99b867436286d683dc57e441fa" member_type: VOTER last_known_addr { host: "127.6.235.126" port: 36441 } }
I20260928 02:13:00.970289 18459 raft_consensus.cc:388] T 00000000000000000000000000000000 P e09f2d99b867436286d683dc57e441fa [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 02:13:00.970476 18459 raft_consensus.cc:482] T 00000000000000000000000000000000 P e09f2d99b867436286d683dc57e441fa [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 02:13:00.970700 18459 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e09f2d99b867436286d683dc57e441fa [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:13:00.972231 18462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:13:00.935567 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44976 (local address 127.6.235.126:36441)
0928 02:13:00.945526 (+ 9959us) server_negotiation.cc:206] Beginning negotiation
0928 02:13:00.945546 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0928 02:13:00.945989 (+ 443us) server_negotiation.cc:407] Connection header received
0928 02:13:00.948729 (+ 2740us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:13:00.948765 (+ 36us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:13:00.949409 (+ 644us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:13:00.949915 (+ 506us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:13:00.951437 (+ 1522us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:13:00.953571 (+ 2134us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:13:00.955398 (+ 1827us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:13:00.956486 (+ 1088us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:13:00.956715 (+ 229us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:13:00.961922 (+ 5207us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:13:00.961934 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:13:00.961956 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:13:00.962010 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:13:00.966166 (+ 4156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:13:00.967145 (+ 979us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:13:00.967154 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:13:00.967163 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:13:00.967372 (+ 209us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:13:00.967814 (+ 442us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:13:00.967818 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:13:00.967821 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:13:00.968586 (+ 765us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:13:00.968856 (+ 270us) server_negotiation.cc:1035] Waiting for connection context
0928 02:13:00.970068 (+ 1212us) server_negotiation.cc:299] Negotiation successful
0928 02:13:00.970563 (+ 495us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9680,"thread_start_us":9439,"threads_started":1}
I20260928 02:13:00.973987 18459 raft_consensus.cc:504] T 00000000000000000000000000000000 P e09f2d99b867436286d683dc57e441fa [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e09f2d99b867436286d683dc57e441fa" member_type: VOTER last_known_addr { host: "127.6.235.126" port: 36441 } }
I20260928 02:13:00.974597 18459 leader_election.cc:302] T 00000000000000000000000000000000 P e09f2d99b867436286d683dc57e441fa [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e09f2d99b867436286d683dc57e441fa; no voters:
I20260928 02:13:00.975256 18459 leader_election.cc:288] T 00000000000000000000000000000000 P e09f2d99b867436286d683dc57e441fa [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 02:13:00.975391 18464 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e09f2d99b867436286d683dc57e441fa [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:13:00.977555 18464 raft_consensus.cc:686] T 00000000000000000000000000000000 P e09f2d99b867436286d683dc57e441fa [term 1 LEADER]: Becoming Leader. State: Replica: e09f2d99b867436286d683dc57e441fa, State: Running, Role: LEADER
I20260928 02:13:00.978223 18464 consensus_queue.cc:237] T 00000000000000000000000000000000 P e09f2d99b867436286d683dc57e441fa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e09f2d99b867436286d683dc57e441fa" member_type: VOTER last_known_addr { host: "127.6.235.126" port: 36441 } }
I20260928 02:13:00.978389 18459 sys_catalog.cc:564] T 00000000000000000000000000000000 P e09f2d99b867436286d683dc57e441fa [sys.catalog]: configured and running, proceeding with master startup.
I20260928 02:13:00.982789 18465 sys_catalog.cc:455] T 00000000000000000000000000000000 P e09f2d99b867436286d683dc57e441fa [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e09f2d99b867436286d683dc57e441fa" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e09f2d99b867436286d683dc57e441fa" member_type: VOTER last_known_addr { host: "127.6.235.126" port: 36441 } } }
I20260928 02:13:00.983144 18465 sys_catalog.cc:458] T 00000000000000000000000000000000 P e09f2d99b867436286d683dc57e441fa [sys.catalog]: This master's current role is: LEADER
I20260928 02:13:00.983613 18466 sys_catalog.cc:455] T 00000000000000000000000000000000 P e09f2d99b867436286d683dc57e441fa [sys.catalog]: SysCatalogTable state changed. Reason: New leader e09f2d99b867436286d683dc57e441fa. Latest consensus state: current_term: 1 leader_uuid: "e09f2d99b867436286d683dc57e441fa" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e09f2d99b867436286d683dc57e441fa" member_type: VOTER last_known_addr { host: "127.6.235.126" port: 36441 } } }
I20260928 02:13:00.983848 18466 sys_catalog.cc:458] T 00000000000000000000000000000000 P e09f2d99b867436286d683dc57e441fa [sys.catalog]: This master's current role is: LEADER
I20260928 02:13:00.987918 18470 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 02:13:00.995642 18470 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 02:13:01.023610 18470 catalog_manager.cc:1347] Generated new cluster ID: 7c7f7f3741764b24bb2c5be7e4f899e1
I20260928 02:13:01.023768 18470 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 02:13:01.041946 18459 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790561432251112-7085-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260928 02:13:01.042588 18470 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 02:13:01.044301 18470 catalog_manager.cc:1504] Loading token signing keys...
I20260928 02:13:01.061311 18470 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e09f2d99b867436286d683dc57e441fa: Generated new TSK 0
I20260928 02:13:01.062546 18470 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.6.235.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.235.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.235.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790561432251112-7085-0/krb5kdc/kudu_127.6.235.65.keytab.
Entry for principal kudu/127.6.235.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790561432251112-7085-0/krb5kdc/kudu_127.6.235.65.keytab.
I20260928 02:13:01.886266 7085 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTlaUuc/build/asan/bin/kudu
/tmp/dist-test-taskTlaUuc/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790561432251112-7085-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790561432251112-7085-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790561432251112-7085-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790561432251112-7085-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.6.235.84:42099/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.235.65:0
--local_ip_for_outbound_sockets=127.6.235.65
--webserver_interface=127.6.235.65
--webserver_port=0
--tserver_master_addrs=127.6.235.126:36441
--keytab_file=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790561432251112-7085-0/krb5kdc/kudu_127.6.235.65.keytab
--principal=kudu/127.6.235.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.235.84:39613
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790561432251112-7085-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790561432251112-7085-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790561432251112-7085-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 28 02:13:02 dist-test-slave-86gv krb5kdc[17948](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790561582, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.235.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260928 02:13:02.363451 18504 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:13:02.363819 18504 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:13:02.363986 18504 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:13:02.373231 18504 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:13:02.373389 18504 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:13:02.373574 18504 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.235.65
I20260928 02:13:02.386229 18504 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.235.84:39613
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790561432251112-7085-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790561432251112-7085-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.6.235.84:42099/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790561432251112-7085-0/krb5kdc/kudu_127.6.235.65.keytab
--principal=kudu/127.6.235.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.235.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790561432251112-7085-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.6.235.65
--webserver_port=0
--tserver_master_addrs=127.6.235.126:36441
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.235.65
--log_dir=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790561432251112-7085-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 00:51:59 UTC on ccycloud.quasar-axgkwn.root.comops.site
build id 4676464
ASAN enabled
I20260928 02:13:02.388206 18504 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:13:02.390133 18504 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:13:02.409955 18514 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:13:02.411114 18515 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:13:02.419674 18517 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:13:02.420125 18504 server_base.cc:1034] running on GCE node
I20260928 02:13:02.420849 18504 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:13:02.422070 18504 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:13:02.423359 18504 hybrid_clock.cc:648] HybridClock initialized: now 1790561582423290 us; error 66 us; skew 500 ppm
I20260928 02:13:02.423744 18504 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 02:13:02 dist-test-slave-86gv krb5kdc[17948](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561582, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.235.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:13:02.427251 18504 init.cc:377] Logged in from keytab as kudu/127.6.235.65@KRBTEST.COM (short username kudu)
I20260928 02:13:02.431202 18504 webserver.cc:460] Webserver started at http://127.6.235.65:33615/ using document root <none> and password file <none>
I20260928 02:13:02.432036 18504 fs_manager.cc:359] Metadata directory not provided
I20260928 02:13:02.432202 18504 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:13:02.432550 18504 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 28 02:13:02 dist-test-slave-86gv krb5kdc[17948](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561582, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.235.65@KRBTEST.COM for HTTP/127.6.235.84@KRBTEST.COM
I20260928 02:13:02.448880 18504 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790561432251112-7085-0/minicluster-data/ts-0/data/instance:
uuid: "222bec4e40b744999bced4a1584e7567"
format_stamp: "Formatted at 2026-09-28 02:13:02 on dist-test-slave-86gv"
server_key: "2a2ed06af2a3884f7b9983bd112f850b"
server_key_iv: "c491ee1a3d6596d4a24ac056ce0a4fbb"
server_key_version: "kuduclusterkey@0"
I20260928 02:13:02.450232 18504 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790561432251112-7085-0/minicluster-data/ts-0/wal/instance:
uuid: "222bec4e40b744999bced4a1584e7567"
format_stamp: "Formatted at 2026-09-28 02:13:02 on dist-test-slave-86gv"
server_key: "2a2ed06af2a3884f7b9983bd112f850b"
server_key_iv: "c491ee1a3d6596d4a24ac056ce0a4fbb"
server_key_version: "kuduclusterkey@0"
I20260928 02:13:02.465833 18504 fs_manager.cc:691] Time spent creating directory manager: real 0.015s user 0.009s sys 0.004s
I20260928 02:13:02.487792 18524 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:13:02.490018 18504 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.001s
I20260928 02:13:02.490243 18504 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790561432251112-7085-0/minicluster-data/ts-0/data,/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790561432251112-7085-0/minicluster-data/ts-0/wal
uuid: "222bec4e40b744999bced4a1584e7567"
format_stamp: "Formatted at 2026-09-28 02:13:02 on dist-test-slave-86gv"
server_key: "2a2ed06af2a3884f7b9983bd112f850b"
server_key_iv: "c491ee1a3d6596d4a24ac056ce0a4fbb"
server_key_version: "kuduclusterkey@0"
I20260928 02:13:02.490532 18504 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790561432251112-7085-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790561432251112-7085-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790561432251112-7085-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:13:02.525764 18504 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:13:02.531733 18504 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:13:02.532194 18504 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:13:02.545369 18504 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:13:02.560456 18504 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:13:02.560670 18504 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.001s sys 0.000s
I20260928 02:13:02.560916 18504 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:13:02.561066 18504 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 28 02:13:02 dist-test-slave-86gv krb5kdc[17948](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561582, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.235.126@KRBTEST.COM for HTTP/127.6.235.84@KRBTEST.COM
I20260928 02:13:03.025852 18504 rpc_server.cc:307] RPC server started. Bound to: 127.6.235.65:38651
I20260928 02:13:03.025873 18636 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.235.65:38651 every 8 connection(s)
I20260928 02:13:03.027302 18504 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790561432251112-7085-0/minicluster-data/ts-0/data/info.pb
I20260928 02:13:03.032526 7085 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTlaUuc/build/asan/bin/kudu as pid 18504
I20260928 02:13:03.032727 7085 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790561432251112-7085-0/minicluster-data/ts-0/wal/instance
I20260928 02:13:03.033577 7085 external_mini_cluster.cc:1428] Setting key 0004fa40d889a26551b3a9973b05af21
Sep 28 02:13:03 dist-test-slave-86gv krb5kdc[17948](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561582, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.235.65@KRBTEST.COM for kudu/127.6.235.126@KRBTEST.COM
I20260928 02:13:03.099737 18640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:13:03.037493 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.235.65:50979 (local address 127.6.235.126:36441)
0928 02:13:03.045388 (+ 7895us) server_negotiation.cc:206] Beginning negotiation
0928 02:13:03.045397 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 02:13:03.072617 (+ 27220us) server_negotiation.cc:407] Connection header received
0928 02:13:03.076967 (+ 4350us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:13:03.076973 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:13:03.077081 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:13:03.077176 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:13:03.081812 (+ 4636us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:13:03.083010 (+ 1198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:13:03.085548 (+ 2538us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:13:03.086194 (+ 646us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:13:03.086297 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:13:03.091207 (+ 4910us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:13:03.091217 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:13:03.091225 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:13:03.091274 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:13:03.095262 (+ 3988us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:13:03.096576 (+ 1314us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:13:03.096585 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:13:03.096591 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:13:03.096746 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:13:03.097483 (+ 737us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:13:03.097490 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:13:03.097496 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:13:03.097890 (+ 394us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:13:03.098029 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0928 02:13:03.099102 (+ 1073us) server_negotiation.cc:299] Negotiation successful
0928 02:13:03.099353 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7734,"thread_start_us":7557,"threads_started":1}
I20260928 02:13:03.102451 18639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:13:03.031620 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.235.126:36441 (local address 127.6.235.65:50979)
0928 02:13:03.072256 (+ 40636us) negotiation.cc:107] Waiting for socket to connect
0928 02:13:03.072450 (+ 194us) client_negotiation.cc:174] Beginning negotiation
0928 02:13:03.076543 (+ 4093us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:13:03.078728 (+ 2185us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:13:03.078744 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:13:03.080557 (+ 1813us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:13:03.081557 (+ 1000us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:13:03.081587 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:13:03.083253 (+ 1666us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:13:03.083259 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:13:03.085366 (+ 2107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:13:03.085381 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:13:03.086673 (+ 1292us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:13:03.086680 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:13:03.086972 (+ 292us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:13:03.087863 (+ 891us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:13:03.087895 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:13:03.090923 (+ 3028us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:13:03.095543 (+ 4620us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:13:03.095556 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:13:03.095578 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:13:03.096387 (+ 809us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:13:03.096975 (+ 588us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:13:03.096983 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:13:03.096989 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:13:03.097267 (+ 278us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:13:03.098126 (+ 859us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:13:03.098137 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:13:03.098655 (+ 518us) client_negotiation.cc:769] Sending connection context
0928 02:13:03.100327 (+ 1672us) client_negotiation.cc:240] Negotiation successful
0928 02:13:03.100921 (+ 594us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":40146,"thread_start_us":43410,"threads_started":1}
I20260928 02:13:03.104800 18637 heartbeater.cc:347] Connected to a master server at 127.6.235.126:36441
I20260928 02:13:03.105186 18637 heartbeater.cc:464] Registering TS with master...
I20260928 02:13:03.106119 18637 heartbeater.cc:511] Master 127.6.235.126:36441 requested a full tablet report, sending...
I20260928 02:13:03.108809 18424 ts_manager.cc:194] Registered new tserver with Master: 222bec4e40b744999bced4a1584e7567 (127.6.235.65:38651)
I20260928 02:13:03.111199 18424 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.235.65@KRBTEST.COM'} at 127.6.235.65:50979
WARNING: no policy specified for kudu/127.6.235.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.235.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.235.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790561432251112-7085-0/krb5kdc/kudu_127.6.235.66.keytab.
Entry for principal kudu/127.6.235.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790561432251112-7085-0/krb5kdc/kudu_127.6.235.66.keytab.
W20260928 02:13:03.152607 7085 mini_kdc.cc:262] Time spent creating service keytab for kudu/127.6.235.66: real 0.112s user 0.000s sys 0.017s
I20260928 02:13:03.153622 7085 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTlaUuc/build/asan/bin/kudu
/tmp/dist-test-taskTlaUuc/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790561432251112-7085-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790561432251112-7085-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790561432251112-7085-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790561432251112-7085-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.6.235.84:42099/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.235.66:0
--local_ip_for_outbound_sockets=127.6.235.66
--webserver_interface=127.6.235.66
--webserver_port=0
--tserver_master_addrs=127.6.235.126:36441
--keytab_file=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790561432251112-7085-0/krb5kdc/kudu_127.6.235.66.keytab
--principal=kudu/127.6.235.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.235.84:39613
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790561432251112-7085-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790561432251112-7085-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790561432251112-7085-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 28 02:13:03 dist-test-slave-86gv krb5kdc[17948](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561582, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.235.126@KRBTEST.COM for HTTP/127.6.235.84@KRBTEST.COM
W20260928 02:13:03.611675 18643 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:13:03.612032 18643 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:13:03.612221 18643 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:13:03.619716 18643 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:13:03.619907 18643 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:13:03.620085 18643 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.235.66
I20260928 02:13:03.630646 18643 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.235.84:39613
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790561432251112-7085-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790561432251112-7085-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.6.235.84:42099/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790561432251112-7085-0/krb5kdc/kudu_127.6.235.66.keytab
--principal=kudu/127.6.235.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.235.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790561432251112-7085-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.6.235.66
--webserver_port=0
--tserver_master_addrs=127.6.235.126:36441
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.235.66
--log_dir=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790561432251112-7085-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 00:51:59 UTC on ccycloud.quasar-axgkwn.root.comops.site
build id 4676464
ASAN enabled
I20260928 02:13:03.632093 18643 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:13:03.633526 18643 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:13:03.653107 18660 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:13:03.653120 18658 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:13:03.653769 18657 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:13:03.654222 18643 server_base.cc:1034] running on GCE node
I20260928 02:13:03.654910 18643 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:13:03.656086 18643 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:13:03.657303 18643 hybrid_clock.cc:648] HybridClock initialized: now 1790561583657248 us; error 48 us; skew 500 ppm
I20260928 02:13:03.657651 18643 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 02:13:03 dist-test-slave-86gv krb5kdc[17948](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561583, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.235.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:13:03.661098 18643 init.cc:377] Logged in from keytab as kudu/127.6.235.66@KRBTEST.COM (short username kudu)
I20260928 02:13:03.662729 18643 webserver.cc:460] Webserver started at http://127.6.235.66:44873/ using document root <none> and password file <none>
I20260928 02:13:03.663741 18643 fs_manager.cc:359] Metadata directory not provided
I20260928 02:13:03.663918 18643 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:13:03.664264 18643 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 28 02:13:03 dist-test-slave-86gv krb5kdc[17948](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561583, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.235.66@KRBTEST.COM for HTTP/127.6.235.84@KRBTEST.COM
I20260928 02:13:03.684010 18643 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790561432251112-7085-0/minicluster-data/ts-1/data/instance:
uuid: "e17eecb4a4fe4e1aaee6b111171975b3"
format_stamp: "Formatted at 2026-09-28 02:13:03 on dist-test-slave-86gv"
server_key: "dac5b39461ed3fd8e7e57cfaedd789a0"
server_key_iv: "b8fddee2b60cf82b3082d4ec14c7732e"
server_key_version: "kuduclusterkey@0"
I20260928 02:13:03.685199 18643 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790561432251112-7085-0/minicluster-data/ts-1/wal/instance:
uuid: "e17eecb4a4fe4e1aaee6b111171975b3"
format_stamp: "Formatted at 2026-09-28 02:13:03 on dist-test-slave-86gv"
server_key: "dac5b39461ed3fd8e7e57cfaedd789a0"
server_key_iv: "b8fddee2b60cf82b3082d4ec14c7732e"
server_key_version: "kuduclusterkey@0"
I20260928 02:13:03.692859 18643 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.006s sys 0.000s
I20260928 02:13:03.709889 18667 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:13:03.711836 18643 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260928 02:13:03.712061 18643 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790561432251112-7085-0/minicluster-data/ts-1/data,/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790561432251112-7085-0/minicluster-data/ts-1/wal
uuid: "e17eecb4a4fe4e1aaee6b111171975b3"
format_stamp: "Formatted at 2026-09-28 02:13:03 on dist-test-slave-86gv"
server_key: "dac5b39461ed3fd8e7e57cfaedd789a0"
server_key_iv: "b8fddee2b60cf82b3082d4ec14c7732e"
server_key_version: "kuduclusterkey@0"
I20260928 02:13:03.712363 18643 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790561432251112-7085-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790561432251112-7085-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790561432251112-7085-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:13:03.731608 18643 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:13:03.736259 18643 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:13:03.736603 18643 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:13:03.737862 18643 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:13:03.739332 18643 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:13:03.739406 18643 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:13:03.739557 18643 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:13:03.739606 18643 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:13:03.775588 18643 rpc_server.cc:307] RPC server started. Bound to: 127.6.235.66:43937
I20260928 02:13:03.775614 18779 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.235.66:43937 every 8 connection(s)
I20260928 02:13:03.777040 18643 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790561432251112-7085-0/minicluster-data/ts-1/data/info.pb
I20260928 02:13:03.780674 7085 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTlaUuc/build/asan/bin/kudu as pid 18643
I20260928 02:13:03.781011 7085 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790561432251112-7085-0/minicluster-data/ts-1/wal/instance
I20260928 02:13:03.781715 7085 external_mini_cluster.cc:1428] Setting key f0ef99be4bc715f2cdcf56d0c7fda38a
Sep 28 02:13:03 dist-test-slave-86gv krb5kdc[17948](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561583, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.235.66@KRBTEST.COM for kudu/127.6.235.126@KRBTEST.COM
I20260928 02:13:03.810063 18782 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:13:03.780092 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.235.66:47089 (local address 127.6.235.126:36441)
0928 02:13:03.793277 (+ 13185us) server_negotiation.cc:206] Beginning negotiation
0928 02:13:03.793283 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 02:13:03.793309 (+ 26us) server_negotiation.cc:407] Connection header received
0928 02:13:03.794960 (+ 1651us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:13:03.794969 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:13:03.795072 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:13:03.795198 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:13:03.797480 (+ 2282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:13:03.798239 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:13:03.799899 (+ 1660us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:13:03.800333 (+ 434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:13:03.800469 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:13:03.804088 (+ 3619us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:13:03.804093 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:13:03.804096 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:13:03.804129 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:13:03.806735 (+ 2606us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:13:03.807496 (+ 761us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:13:03.807509 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:13:03.807512 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:13:03.807631 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:13:03.808084 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:13:03.808090 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:13:03.808093 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:13:03.808467 (+ 374us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:13:03.808628 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0928 02:13:03.809434 (+ 806us) server_negotiation.cc:299] Negotiation successful
0928 02:13:03.809665 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13063,"thread_start_us":12953,"threads_started":1}
I20260928 02:13:03.811154 18783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:13:03.780944 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.235.126:36441 (local address 127.6.235.66:47089)
0928 02:13:03.791605 (+ 10661us) negotiation.cc:107] Waiting for socket to connect
0928 02:13:03.791666 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0928 02:13:03.794581 (+ 2915us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:13:03.795539 (+ 958us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:13:03.795552 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:13:03.796183 (+ 631us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:13:03.797143 (+ 960us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:13:03.797171 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:13:03.798407 (+ 1236us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:13:03.798412 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:13:03.799722 (+ 1310us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:13:03.799735 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:13:03.800582 (+ 847us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:13:03.800589 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:13:03.800834 (+ 245us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:13:03.801645 (+ 811us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:13:03.801682 (+ 37us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:13:03.803841 (+ 2159us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:13:03.806886 (+ 3045us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:13:03.806896 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:13:03.806915 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:13:03.807360 (+ 445us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:13:03.807742 (+ 382us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:13:03.807748 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:13:03.807751 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:13:03.807957 (+ 206us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:13:03.808603 (+ 646us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:13:03.808612 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:13:03.809033 (+ 421us) client_negotiation.cc:769] Sending connection context
0928 02:13:03.809336 (+ 303us) client_negotiation.cc:240] Negotiation successful
0928 02:13:03.809881 (+ 545us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9252,"thread_start_us":9266,"threads_started":1}
I20260928 02:13:03.812721 18780 heartbeater.cc:347] Connected to a master server at 127.6.235.126:36441
I20260928 02:13:03.813082 18780 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.6.235.67@KRBTEST.COM; defaulting to no policy
I20260928 02:13:03.813910 18780 heartbeater.cc:511] Master 127.6.235.126:36441 requested a full tablet report, sending...
I20260928 02:13:03.815634 18424 ts_manager.cc:194] Registered new tserver with Master: e17eecb4a4fe4e1aaee6b111171975b3 (127.6.235.66:43937)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.235.67@KRBTEST.COM" created.
I20260928 02:13:03.816658 18424 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.235.66@KRBTEST.COM'} at 127.6.235.66:47089
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.235.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790561432251112-7085-0/krb5kdc/kudu_127.6.235.67.keytab.
Entry for principal kudu/127.6.235.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790561432251112-7085-0/krb5kdc/kudu_127.6.235.67.keytab.
I20260928 02:13:03.853354 7085 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTlaUuc/build/asan/bin/kudu
/tmp/dist-test-taskTlaUuc/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790561432251112-7085-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790561432251112-7085-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790561432251112-7085-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790561432251112-7085-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.6.235.84:42099/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.235.67:0
--local_ip_for_outbound_sockets=127.6.235.67
--webserver_interface=127.6.235.67
--webserver_port=0
--tserver_master_addrs=127.6.235.126:36441
--keytab_file=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790561432251112-7085-0/krb5kdc/kudu_127.6.235.67.keytab
--principal=kudu/127.6.235.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.235.84:39613
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790561432251112-7085-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790561432251112-7085-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790561432251112-7085-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 02:13:04.115676 18637 heartbeater.cc:503] Master 127.6.235.126:36441 was elected leader, sending a full tablet report...
W20260928 02:13:04.222893 18786 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:13:04.223284 18786 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:13:04.223443 18786 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:13:04.230901 18786 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:13:04.231058 18786 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:13:04.231247 18786 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.235.67
I20260928 02:13:04.241580 18786 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.235.84:39613
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790561432251112-7085-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790561432251112-7085-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.6.235.84:42099/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790561432251112-7085-0/krb5kdc/kudu_127.6.235.67.keytab
--principal=kudu/127.6.235.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.235.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790561432251112-7085-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.6.235.67
--webserver_port=0
--tserver_master_addrs=127.6.235.126:36441
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.235.67
--log_dir=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790561432251112-7085-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 00:51:59 UTC on ccycloud.quasar-axgkwn.root.comops.site
build id 4676464
ASAN enabled
I20260928 02:13:04.243005 18786 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:13:04.244377 18786 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:13:04.258090 18794 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:13:04.258112 18792 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:13:04.258399 18791 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:13:04.259371 18786 server_base.cc:1034] running on GCE node
I20260928 02:13:04.259984 18786 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:13:04.261015 18786 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:13:04.262236 18786 hybrid_clock.cc:648] HybridClock initialized: now 1790561584262191 us; error 43 us; skew 500 ppm
I20260928 02:13:04.262544 18786 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 02:13:04 dist-test-slave-86gv krb5kdc[17948](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561584, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.235.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:13:04.265784 18786 init.cc:377] Logged in from keytab as kudu/127.6.235.67@KRBTEST.COM (short username kudu)
I20260928 02:13:04.267279 18786 webserver.cc:460] Webserver started at http://127.6.235.67:34429/ using document root <none> and password file <none>
I20260928 02:13:04.267998 18786 fs_manager.cc:359] Metadata directory not provided
I20260928 02:13:04.268102 18786 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:13:04.268373 18786 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 28 02:13:04 dist-test-slave-86gv krb5kdc[17948](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561584, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.235.67@KRBTEST.COM for HTTP/127.6.235.84@KRBTEST.COM
I20260928 02:13:04.285171 18786 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790561432251112-7085-0/minicluster-data/ts-2/data/instance:
uuid: "fd994f1fa3b2442f91a8df65847891f6"
format_stamp: "Formatted at 2026-09-28 02:13:04 on dist-test-slave-86gv"
server_key: "0854b15e3a8c5d5cf123bf9221770e4a"
server_key_iv: "6e3bd56e606665e02bb700fe8b889568"
server_key_version: "kuduclusterkey@0"
I20260928 02:13:04.286300 18786 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790561432251112-7085-0/minicluster-data/ts-2/wal/instance:
uuid: "fd994f1fa3b2442f91a8df65847891f6"
format_stamp: "Formatted at 2026-09-28 02:13:04 on dist-test-slave-86gv"
server_key: "0854b15e3a8c5d5cf123bf9221770e4a"
server_key_iv: "6e3bd56e606665e02bb700fe8b889568"
server_key_version: "kuduclusterkey@0"
I20260928 02:13:04.293339 18786 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.007s sys 0.000s
I20260928 02:13:04.308826 18801 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:13:04.310604 18786 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.000s sys 0.003s
I20260928 02:13:04.310815 18786 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790561432251112-7085-0/minicluster-data/ts-2/data,/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790561432251112-7085-0/minicluster-data/ts-2/wal
uuid: "fd994f1fa3b2442f91a8df65847891f6"
format_stamp: "Formatted at 2026-09-28 02:13:04 on dist-test-slave-86gv"
server_key: "0854b15e3a8c5d5cf123bf9221770e4a"
server_key_iv: "6e3bd56e606665e02bb700fe8b889568"
server_key_version: "kuduclusterkey@0"
I20260928 02:13:04.311072 18786 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790561432251112-7085-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790561432251112-7085-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790561432251112-7085-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:13:04.352537 18786 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:13:04.356863 18786 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:13:04.357226 18786 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:13:04.358217 18786 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:13:04.359645 18786 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:13:04.359716 18786 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:13:04.359841 18786 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:13:04.359891 18786 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:13:04.394562 18786 rpc_server.cc:307] RPC server started. Bound to: 127.6.235.67:34991
I20260928 02:13:04.394603 18913 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.235.67:34991 every 8 connection(s)
I20260928 02:13:04.396071 18786 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790561432251112-7085-0/minicluster-data/ts-2/data/info.pb
I20260928 02:13:04.403611 7085 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTlaUuc/build/asan/bin/kudu as pid 18786
I20260928 02:13:04.403828 7085 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790561432251112-7085-0/minicluster-data/ts-2/wal/instance
I20260928 02:13:04.404387 7085 external_mini_cluster.cc:1428] Setting key 227e9b7410a67776db0995b80b5d2460
Sep 28 02:13:04 dist-test-slave-86gv krb5kdc[17948](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561584, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.235.67@KRBTEST.COM for kudu/127.6.235.126@KRBTEST.COM
I20260928 02:13:04.417371 18916 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:13:04.398755 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.235.67:44257 (local address 127.6.235.126:36441)
0928 02:13:04.399189 (+ 434us) server_negotiation.cc:206] Beginning negotiation
0928 02:13:04.399195 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 02:13:04.400284 (+ 1089us) server_negotiation.cc:407] Connection header received
0928 02:13:04.402085 (+ 1801us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:13:04.402090 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:13:04.402170 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:13:04.402275 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:13:04.403712 (+ 1437us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:13:04.404810 (+ 1098us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:13:04.406718 (+ 1908us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:13:04.407379 (+ 661us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:13:04.407533 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:13:04.410992 (+ 3459us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:13:04.410997 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:13:04.411000 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:13:04.411034 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:13:04.413909 (+ 2875us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:13:04.414734 (+ 825us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:13:04.414747 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:13:04.414749 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:13:04.414885 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:13:04.415464 (+ 579us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:13:04.415481 (+ 17us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:13:04.415488 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:13:04.415828 (+ 340us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:13:04.416007 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0928 02:13:04.416757 (+ 750us) server_negotiation.cc:299] Negotiation successful
0928 02:13:04.417042 (+ 285us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":314,"thread_start_us":206,"threads_started":1}
I20260928 02:13:04.418596 18917 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:13:04.399432 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.235.126:36441 (local address 127.6.235.67:44257)
0928 02:13:04.400121 (+ 689us) negotiation.cc:107] Waiting for socket to connect
0928 02:13:04.400164 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0928 02:13:04.401824 (+ 1660us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:13:04.402521 (+ 697us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:13:04.402529 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:13:04.402952 (+ 423us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:13:04.403533 (+ 581us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:13:04.403549 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:13:04.405021 (+ 1472us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:13:04.405027 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:13:04.406579 (+ 1552us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:13:04.406591 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:13:04.407622 (+ 1031us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:13:04.407632 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:13:04.407887 (+ 255us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:13:04.408635 (+ 748us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:13:04.408659 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:13:04.410727 (+ 2068us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:13:04.414082 (+ 3355us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:13:04.414092 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:13:04.414113 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:13:04.414579 (+ 466us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:13:04.415011 (+ 432us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:13:04.415016 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:13:04.415019 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:13:04.415292 (+ 273us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:13:04.416033 (+ 741us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:13:04.416040 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:13:04.416397 (+ 357us) client_negotiation.cc:769] Sending connection context
0928 02:13:04.416676 (+ 279us) client_negotiation.cc:240] Negotiation successful
0928 02:13:04.417218 (+ 542us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":457,"thread_start_us":272,"threads_started":1}
I20260928 02:13:04.420445 18914 heartbeater.cc:347] Connected to a master server at 127.6.235.126:36441
I20260928 02:13:04.420753 18914 heartbeater.cc:464] Registering TS with master...
I20260928 02:13:04.421574 18914 heartbeater.cc:511] Master 127.6.235.126:36441 requested a full tablet report, sending...
I20260928 02:13:04.423070 18423 ts_manager.cc:194] Registered new tserver with Master: fd994f1fa3b2442f91a8df65847891f6 (127.6.235.67:34991)
I20260928 02:13:04.424023 18423 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.235.67@KRBTEST.COM'} at 127.6.235.67:44257
I20260928 02:13:04.432777 7085 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 02:13:04.450465 18916 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:13:04.437697 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48296 (local address 127.6.235.126:36441)
0928 02:13:04.437935 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0928 02:13:04.437942 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 02:13:04.438081 (+ 139us) server_negotiation.cc:407] Connection header received
0928 02:13:04.438339 (+ 258us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:13:04.438344 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:13:04.438440 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:13:04.438556 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:13:04.440000 (+ 1444us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:13:04.440717 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:13:04.442054 (+ 1337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:13:04.442509 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:13:04.442650 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:13:04.445125 (+ 2475us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:13:04.445130 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:13:04.445133 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:13:04.445167 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:13:04.447649 (+ 2482us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:13:04.448415 (+ 766us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:13:04.448427 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:13:04.448430 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:13:04.448570 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:13:04.448996 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:13:04.449000 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:13:04.449002 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:13:04.449283 (+ 281us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:13:04.449555 (+ 272us) server_negotiation.cc:1035] Waiting for connection context
0928 02:13:04.449887 (+ 332us) server_negotiation.cc:299] Negotiation successful
0928 02:13:04.450135 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
I20260928 02:13:04.454923 18423 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48296:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 02:13:04.462306 18423 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 02:13:04.500183 18927 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:13:04.475469 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.235.65:38651 (local address 127.0.0.1:59364)
0928 02:13:04.488853 (+ 13384us) negotiation.cc:107] Waiting for socket to connect
0928 02:13:04.488869 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 02:13:04.489069 (+ 200us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:13:04.492488 (+ 3419us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:13:04.492498 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:13:04.492537 (+ 39us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:13:04.493014 (+ 477us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:13:04.493029 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:13:04.494986 (+ 1957us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:13:04.494993 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:13:04.496760 (+ 1767us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:13:04.496781 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:13:04.499182 (+ 2401us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:13:04.499191 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:13:04.499418 (+ 227us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:13:04.499475 (+ 57us) client_negotiation.cc:769] Sending connection context
0928 02:13:04.499610 (+ 135us) client_negotiation.cc:240] Negotiation successful
0928 02:13:04.499811 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13267,"thread_start_us":11985,"threads_started":1}
I20260928 02:13:04.500247 18930 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:13:04.475569 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59364 (local address 127.6.235.65:38651)
0928 02:13:04.491422 (+ 15853us) server_negotiation.cc:206] Beginning negotiation
0928 02:13:04.491436 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0928 02:13:04.491477 (+ 41us) server_negotiation.cc:407] Connection header received
0928 02:13:04.491617 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:13:04.491630 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:13:04.492015 (+ 385us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:13:04.492323 (+ 308us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:13:04.493249 (+ 926us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:13:04.494815 (+ 1566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:13:04.497444 (+ 2629us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:13:04.498985 (+ 1541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:13:04.499113 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:13:04.499375 (+ 262us) server_negotiation.cc:1035] Waiting for connection context
0928 02:13:04.499708 (+ 333us) server_negotiation.cc:299] Negotiation successful
0928 02:13:04.499885 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15599,"thread_start_us":15386,"threads_started":1}
I20260928 02:13:04.501641 18931 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:13:04.478222 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.235.67:34991 (local address 127.0.0.1:39178)
0928 02:13:04.492435 (+ 14213us) negotiation.cc:107] Waiting for socket to connect
0928 02:13:04.492453 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0928 02:13:04.492578 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:13:04.494147 (+ 1569us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:13:04.494154 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:13:04.494184 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:13:04.494708 (+ 524us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:13:04.494724 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:13:04.496802 (+ 2078us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:13:04.496807 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:13:04.498873 (+ 2066us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:13:04.498886 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:13:04.500714 (+ 1828us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:13:04.500721 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:13:04.500888 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:13:04.500948 (+ 60us) client_negotiation.cc:769] Sending connection context
0928 02:13:04.501112 (+ 164us) client_negotiation.cc:240] Negotiation successful
0928 02:13:04.501232 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14057,"mutex_wait_us":280,"thread_start_us":13914,"threads_started":1}
I20260928 02:13:04.502488 18932 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:13:04.478922 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39178 (local address 127.6.235.67:34991)
0928 02:13:04.491894 (+ 12972us) server_negotiation.cc:206] Beginning negotiation
0928 02:13:04.491907 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0928 02:13:04.492663 (+ 756us) server_negotiation.cc:407] Connection header received
0928 02:13:04.492793 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:13:04.492804 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:13:04.493176 (+ 372us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:13:04.494098 (+ 922us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:13:04.494985 (+ 887us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:13:04.496617 (+ 1632us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:13:04.499039 (+ 2422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:13:04.500514 (+ 1475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:13:04.500659 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:13:04.500879 (+ 220us) server_negotiation.cc:1035] Waiting for connection context
0928 02:13:04.501990 (+ 1111us) server_negotiation.cc:299] Negotiation successful
0928 02:13:04.502173 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12745,"thread_start_us":12586,"threads_started":1}
I20260928 02:13:04.502803 18929 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:13:04.475169 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.235.66:43937 (local address 127.0.0.1:59862)
0928 02:13:04.488627 (+ 13458us) negotiation.cc:107] Waiting for socket to connect
0928 02:13:04.488685 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0928 02:13:04.489480 (+ 795us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:13:04.494398 (+ 4918us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:13:04.494404 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:13:04.494434 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:13:04.494875 (+ 441us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:13:04.494887 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:13:04.496886 (+ 1999us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:13:04.496921 (+ 35us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:13:04.499310 (+ 2389us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:13:04.499325 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:13:04.502029 (+ 2704us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:13:04.502036 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:13:04.502208 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:13:04.502237 (+ 29us) client_negotiation.cc:769] Sending connection context
0928 02:13:04.502317 (+ 80us) client_negotiation.cc:240] Negotiation successful
0928 02:13:04.502448 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12499,"mutex_wait_us":1,"thread_start_us":14106,"threads_started":1}
I20260928 02:13:04.503612 18928 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:13:04.475268 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59862 (local address 127.6.235.66:43937)
0928 02:13:04.493593 (+ 18325us) server_negotiation.cc:206] Beginning negotiation
0928 02:13:04.493605 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0928 02:13:04.493633 (+ 28us) server_negotiation.cc:407] Connection header received
0928 02:13:04.493725 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:13:04.493731 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:13:04.494019 (+ 288us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:13:04.494206 (+ 187us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:13:04.495124 (+ 918us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:13:04.496724 (+ 1600us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:13:04.499488 (+ 2764us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:13:04.501334 (+ 1846us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:13:04.501499 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:13:04.501683 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0928 02:13:04.503135 (+ 1452us) server_negotiation.cc:299] Negotiation successful
0928 02:13:04.503300 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18149,"thread_start_us":18026,"threads_started":1}
I20260928 02:13:04.504880 18572 tablet_service.cc:1467] Processing CreateTablet for tablet b6467673a36449ce9d4d2aa649b75d4e (DEFAULT_TABLE table=test-table [id=cfc4a9201edd43fa93b63e3717abbc86]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:13:04.506379 18572 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b6467673a36449ce9d4d2aa649b75d4e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:13:04.506064 18849 tablet_service.cc:1467] Processing CreateTablet for tablet b6467673a36449ce9d4d2aa649b75d4e (DEFAULT_TABLE table=test-table [id=cfc4a9201edd43fa93b63e3717abbc86]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:13:04.507017 18715 tablet_service.cc:1467] Processing CreateTablet for tablet b6467673a36449ce9d4d2aa649b75d4e (DEFAULT_TABLE table=test-table [id=cfc4a9201edd43fa93b63e3717abbc86]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:13:04.507956 18849 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b6467673a36449ce9d4d2aa649b75d4e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:13:04.508376 18715 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b6467673a36449ce9d4d2aa649b75d4e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:13:04.518911 18933 tablet_bootstrap.cc:492] T b6467673a36449ce9d4d2aa649b75d4e P 222bec4e40b744999bced4a1584e7567: Bootstrap starting.
I20260928 02:13:04.522938 18933 tablet_bootstrap.cc:654] T b6467673a36449ce9d4d2aa649b75d4e P 222bec4e40b744999bced4a1584e7567: Neither blocks nor log segments found. Creating new log.
I20260928 02:13:04.524041 18934 tablet_bootstrap.cc:492] T b6467673a36449ce9d4d2aa649b75d4e P e17eecb4a4fe4e1aaee6b111171975b3: Bootstrap starting.
I20260928 02:13:04.524559 18933 log.cc:824] T b6467673a36449ce9d4d2aa649b75d4e P 222bec4e40b744999bced4a1584e7567: Log is configured to *not* fsync() on all Append() calls
I20260928 02:13:04.525882 18935 tablet_bootstrap.cc:492] T b6467673a36449ce9d4d2aa649b75d4e P fd994f1fa3b2442f91a8df65847891f6: Bootstrap starting.
I20260928 02:13:04.528968 18935 tablet_bootstrap.cc:654] T b6467673a36449ce9d4d2aa649b75d4e P fd994f1fa3b2442f91a8df65847891f6: Neither blocks nor log segments found. Creating new log.
I20260928 02:13:04.529327 18934 tablet_bootstrap.cc:654] T b6467673a36449ce9d4d2aa649b75d4e P e17eecb4a4fe4e1aaee6b111171975b3: Neither blocks nor log segments found. Creating new log.
I20260928 02:13:04.530364 18935 log.cc:824] T b6467673a36449ce9d4d2aa649b75d4e P fd994f1fa3b2442f91a8df65847891f6: Log is configured to *not* fsync() on all Append() calls
I20260928 02:13:04.530745 18934 log.cc:824] T b6467673a36449ce9d4d2aa649b75d4e P e17eecb4a4fe4e1aaee6b111171975b3: Log is configured to *not* fsync() on all Append() calls
I20260928 02:13:04.531827 18933 tablet_bootstrap.cc:492] T b6467673a36449ce9d4d2aa649b75d4e P 222bec4e40b744999bced4a1584e7567: No bootstrap required, opened a new log
I20260928 02:13:04.532168 18933 ts_tablet_manager.cc:1397] T b6467673a36449ce9d4d2aa649b75d4e P 222bec4e40b744999bced4a1584e7567: Time spent bootstrapping tablet: real 0.013s user 0.010s sys 0.000s
I20260928 02:13:04.533793 18935 tablet_bootstrap.cc:492] T b6467673a36449ce9d4d2aa649b75d4e P fd994f1fa3b2442f91a8df65847891f6: No bootstrap required, opened a new log
I20260928 02:13:04.533887 18934 tablet_bootstrap.cc:492] T b6467673a36449ce9d4d2aa649b75d4e P e17eecb4a4fe4e1aaee6b111171975b3: No bootstrap required, opened a new log
I20260928 02:13:04.534106 18934 ts_tablet_manager.cc:1397] T b6467673a36449ce9d4d2aa649b75d4e P e17eecb4a4fe4e1aaee6b111171975b3: Time spent bootstrapping tablet: real 0.010s user 0.008s sys 0.000s
I20260928 02:13:04.534132 18935 ts_tablet_manager.cc:1397] T b6467673a36449ce9d4d2aa649b75d4e P fd994f1fa3b2442f91a8df65847891f6: Time spent bootstrapping tablet: real 0.008s user 0.007s sys 0.000s
I20260928 02:13:04.539943 18933 raft_consensus.cc:348] T b6467673a36449ce9d4d2aa649b75d4e P 222bec4e40b744999bced4a1584e7567 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e17eecb4a4fe4e1aaee6b111171975b3" member_type: VOTER last_known_addr { host: "127.6.235.66" port: 43937 } } peers { permanent_uuid: "222bec4e40b744999bced4a1584e7567" member_type: VOTER last_known_addr { host: "127.6.235.65" port: 38651 } } peers { permanent_uuid: "fd994f1fa3b2442f91a8df65847891f6" member_type: VOTER last_known_addr { host: "127.6.235.67" port: 34991 } }
I20260928 02:13:04.540457 18933 raft_consensus.cc:374] T b6467673a36449ce9d4d2aa649b75d4e P 222bec4e40b744999bced4a1584e7567 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:13:04.540563 18933 raft_consensus.cc:729] T b6467673a36449ce9d4d2aa649b75d4e P 222bec4e40b744999bced4a1584e7567 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 222bec4e40b744999bced4a1584e7567, State: Initialized, Role: FOLLOWER
I20260928 02:13:04.541383 18933 consensus_queue.cc:260] T b6467673a36449ce9d4d2aa649b75d4e P 222bec4e40b744999bced4a1584e7567 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e17eecb4a4fe4e1aaee6b111171975b3" member_type: VOTER last_known_addr { host: "127.6.235.66" port: 43937 } } peers { permanent_uuid: "222bec4e40b744999bced4a1584e7567" member_type: VOTER last_known_addr { host: "127.6.235.65" port: 38651 } } peers { permanent_uuid: "fd994f1fa3b2442f91a8df65847891f6" member_type: VOTER last_known_addr { host: "127.6.235.67" port: 34991 } }
I20260928 02:13:04.543118 18933 ts_tablet_manager.cc:1428] T b6467673a36449ce9d4d2aa649b75d4e P 222bec4e40b744999bced4a1584e7567: Time spent starting tablet: real 0.011s user 0.012s sys 0.000s
I20260928 02:13:04.545171 18934 raft_consensus.cc:348] T b6467673a36449ce9d4d2aa649b75d4e P e17eecb4a4fe4e1aaee6b111171975b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e17eecb4a4fe4e1aaee6b111171975b3" member_type: VOTER last_known_addr { host: "127.6.235.66" port: 43937 } } peers { permanent_uuid: "222bec4e40b744999bced4a1584e7567" member_type: VOTER last_known_addr { host: "127.6.235.65" port: 38651 } } peers { permanent_uuid: "fd994f1fa3b2442f91a8df65847891f6" member_type: VOTER last_known_addr { host: "127.6.235.67" port: 34991 } }
I20260928 02:13:04.545573 18934 raft_consensus.cc:374] T b6467673a36449ce9d4d2aa649b75d4e P e17eecb4a4fe4e1aaee6b111171975b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:13:04.545674 18934 raft_consensus.cc:729] T b6467673a36449ce9d4d2aa649b75d4e P e17eecb4a4fe4e1aaee6b111171975b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e17eecb4a4fe4e1aaee6b111171975b3, State: Initialized, Role: FOLLOWER
I20260928 02:13:04.545557 18935 raft_consensus.cc:348] T b6467673a36449ce9d4d2aa649b75d4e P fd994f1fa3b2442f91a8df65847891f6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e17eecb4a4fe4e1aaee6b111171975b3" member_type: VOTER last_known_addr { host: "127.6.235.66" port: 43937 } } peers { permanent_uuid: "222bec4e40b744999bced4a1584e7567" member_type: VOTER last_known_addr { host: "127.6.235.65" port: 38651 } } peers { permanent_uuid: "fd994f1fa3b2442f91a8df65847891f6" member_type: VOTER last_known_addr { host: "127.6.235.67" port: 34991 } }
I20260928 02:13:04.546015 18935 raft_consensus.cc:374] T b6467673a36449ce9d4d2aa649b75d4e P fd994f1fa3b2442f91a8df65847891f6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:13:04.546113 18935 raft_consensus.cc:729] T b6467673a36449ce9d4d2aa649b75d4e P fd994f1fa3b2442f91a8df65847891f6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fd994f1fa3b2442f91a8df65847891f6, State: Initialized, Role: FOLLOWER
I20260928 02:13:04.546319 18934 consensus_queue.cc:260] T b6467673a36449ce9d4d2aa649b75d4e P e17eecb4a4fe4e1aaee6b111171975b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e17eecb4a4fe4e1aaee6b111171975b3" member_type: VOTER last_known_addr { host: "127.6.235.66" port: 43937 } } peers { permanent_uuid: "222bec4e40b744999bced4a1584e7567" member_type: VOTER last_known_addr { host: "127.6.235.65" port: 38651 } } peers { permanent_uuid: "fd994f1fa3b2442f91a8df65847891f6" member_type: VOTER last_known_addr { host: "127.6.235.67" port: 34991 } }
I20260928 02:13:04.546710 18935 consensus_queue.cc:260] T b6467673a36449ce9d4d2aa649b75d4e P fd994f1fa3b2442f91a8df65847891f6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e17eecb4a4fe4e1aaee6b111171975b3" member_type: VOTER last_known_addr { host: "127.6.235.66" port: 43937 } } peers { permanent_uuid: "222bec4e40b744999bced4a1584e7567" member_type: VOTER last_known_addr { host: "127.6.235.65" port: 38651 } } peers { permanent_uuid: "fd994f1fa3b2442f91a8df65847891f6" member_type: VOTER last_known_addr { host: "127.6.235.67" port: 34991 } }
I20260928 02:13:04.548538 18780 heartbeater.cc:503] Master 127.6.235.126:36441 was elected leader, sending a full tablet report...
I20260928 02:13:04.549278 18934 ts_tablet_manager.cc:1428] T b6467673a36449ce9d4d2aa649b75d4e P e17eecb4a4fe4e1aaee6b111171975b3: Time spent starting tablet: real 0.015s user 0.015s sys 0.000s
I20260928 02:13:04.550743 18914 heartbeater.cc:503] Master 127.6.235.126:36441 was elected leader, sending a full tablet report...
I20260928 02:13:04.551393 18935 ts_tablet_manager.cc:1428] T b6467673a36449ce9d4d2aa649b75d4e P fd994f1fa3b2442f91a8df65847891f6: Time spent starting tablet: real 0.017s user 0.016s sys 0.000s
W20260928 02:13:04.581801 18638 tablet.cc:2369] T b6467673a36449ce9d4d2aa649b75d4e P 222bec4e40b744999bced4a1584e7567: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 02:13:04.647823 18915 tablet.cc:2369] T b6467673a36449ce9d4d2aa649b75d4e P fd994f1fa3b2442f91a8df65847891f6: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 02:13:04.779302 18781 tablet.cc:2369] T b6467673a36449ce9d4d2aa649b75d4e P e17eecb4a4fe4e1aaee6b111171975b3: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 02:13:04.817950 18941 raft_consensus.cc:482] T b6467673a36449ce9d4d2aa649b75d4e P fd994f1fa3b2442f91a8df65847891f6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 02:13:04.818293 18941 raft_consensus.cc:504] T b6467673a36449ce9d4d2aa649b75d4e P fd994f1fa3b2442f91a8df65847891f6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e17eecb4a4fe4e1aaee6b111171975b3" member_type: VOTER last_known_addr { host: "127.6.235.66" port: 43937 } } peers { permanent_uuid: "222bec4e40b744999bced4a1584e7567" member_type: VOTER last_known_addr { host: "127.6.235.65" port: 38651 } } peers { permanent_uuid: "fd994f1fa3b2442f91a8df65847891f6" member_type: VOTER last_known_addr { host: "127.6.235.67" port: 34991 } }
I20260928 02:13:04.820035 18941 leader_election.cc:288] T b6467673a36449ce9d4d2aa649b75d4e P fd994f1fa3b2442f91a8df65847891f6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e17eecb4a4fe4e1aaee6b111171975b3 (127.6.235.66:43937), 222bec4e40b744999bced4a1584e7567 (127.6.235.65:38651)
I20260928 02:13:04.826858 18930 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:13:04.820583 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.235.67:55143 (local address 127.6.235.65:38651)
0928 02:13:04.820877 (+ 294us) server_negotiation.cc:206] Beginning negotiation
0928 02:13:04.820882 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 02:13:04.820946 (+ 64us) server_negotiation.cc:407] Connection header received
0928 02:13:04.821173 (+ 227us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:13:04.821181 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:13:04.821271 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:13:04.821379 (+ 108us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:13:04.821934 (+ 555us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:13:04.822951 (+ 1017us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:13:04.824882 (+ 1931us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:13:04.826009 (+ 1127us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:13:04.826134 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:13:04.826188 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0928 02:13:04.826531 (+ 343us) server_negotiation.cc:299] Negotiation successful
0928 02:13:04.826626 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":166}
I20260928 02:13:04.826934 18917 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:13:04.820502 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.235.65:38651 (local address 127.6.235.67:55143)
0928 02:13:04.820792 (+ 290us) negotiation.cc:107] Waiting for socket to connect
0928 02:13:04.820811 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0928 02:13:04.820987 (+ 176us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:13:04.821425 (+ 438us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:13:04.821428 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:13:04.821453 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:13:04.821766 (+ 313us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:13:04.821776 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:13:04.823095 (+ 1319us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:13:04.823101 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:13:04.824726 (+ 1625us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:13:04.824738 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:13:04.826159 (+ 1421us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:13:04.826166 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:13:04.826351 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:13:04.826389 (+ 38us) client_negotiation.cc:769] Sending connection context
0928 02:13:04.826501 (+ 112us) client_negotiation.cc:240] Negotiation successful
0928 02:13:04.826647 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":157}
I20260928 02:13:04.827736 18942 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:13:04.820809 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.235.66:43937 (local address 127.6.235.67:42265)
0928 02:13:04.821450 (+ 641us) negotiation.cc:107] Waiting for socket to connect
0928 02:13:04.821468 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0928 02:13:04.821606 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:13:04.821993 (+ 387us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:13:04.822000 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:13:04.822028 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:13:04.822389 (+ 361us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:13:04.822399 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:13:04.823568 (+ 1169us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:13:04.823571 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:13:04.825518 (+ 1947us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:13:04.825533 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:13:04.826972 (+ 1439us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:13:04.826981 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:13:04.827156 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:13:04.827212 (+ 56us) client_negotiation.cc:769] Sending connection context
0928 02:13:04.827314 (+ 102us) client_negotiation.cc:240] Negotiation successful
0928 02:13:04.827441 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":524,"thread_start_us":351,"threads_started":1}
I20260928 02:13:04.827761 18928 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:13:04.820480 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.235.67:42265 (local address 127.6.235.66:43937)
0928 02:13:04.820712 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0928 02:13:04.820719 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 02:13:04.821668 (+ 949us) server_negotiation.cc:407] Connection header received
0928 02:13:04.821757 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:13:04.821762 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:13:04.821844 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:13:04.821938 (+ 94us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:13:04.822551 (+ 613us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:13:04.823420 (+ 869us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:13:04.825691 (+ 2271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:13:04.826749 (+ 1058us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:13:04.826889 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:13:04.826980 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0928 02:13:04.827355 (+ 375us) server_negotiation.cc:299] Negotiation successful
0928 02:13:04.827496 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260928 02:13:04.828282 18592 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b6467673a36449ce9d4d2aa649b75d4e" candidate_uuid: "fd994f1fa3b2442f91a8df65847891f6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "222bec4e40b744999bced4a1584e7567" is_pre_election: true
I20260928 02:13:04.828938 18592 raft_consensus.cc:2454] T b6467673a36449ce9d4d2aa649b75d4e P 222bec4e40b744999bced4a1584e7567 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fd994f1fa3b2442f91a8df65847891f6 in term 0.
I20260928 02:13:04.829030 18735 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b6467673a36449ce9d4d2aa649b75d4e" candidate_uuid: "fd994f1fa3b2442f91a8df65847891f6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e17eecb4a4fe4e1aaee6b111171975b3" is_pre_election: true
I20260928 02:13:04.829350 18940 raft_consensus.cc:482] T b6467673a36449ce9d4d2aa649b75d4e P e17eecb4a4fe4e1aaee6b111171975b3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 02:13:04.829658 18940 raft_consensus.cc:504] T b6467673a36449ce9d4d2aa649b75d4e P e17eecb4a4fe4e1aaee6b111171975b3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e17eecb4a4fe4e1aaee6b111171975b3" member_type: VOTER last_known_addr { host: "127.6.235.66" port: 43937 } } peers { permanent_uuid: "222bec4e40b744999bced4a1584e7567" member_type: VOTER last_known_addr { host: "127.6.235.65" port: 38651 } } peers { permanent_uuid: "fd994f1fa3b2442f91a8df65847891f6" member_type: VOTER last_known_addr { host: "127.6.235.67" port: 34991 } }
I20260928 02:13:04.829828 18805 leader_election.cc:302] T b6467673a36449ce9d4d2aa649b75d4e P fd994f1fa3b2442f91a8df65847891f6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 222bec4e40b744999bced4a1584e7567, fd994f1fa3b2442f91a8df65847891f6; no voters:
I20260928 02:13:04.830231 18941 raft_consensus.cc:2781] T b6467673a36449ce9d4d2aa649b75d4e P fd994f1fa3b2442f91a8df65847891f6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 02:13:04.830269 18735 raft_consensus.cc:2454] T b6467673a36449ce9d4d2aa649b75d4e P e17eecb4a4fe4e1aaee6b111171975b3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fd994f1fa3b2442f91a8df65847891f6 in term 0.
I20260928 02:13:04.830371 18941 raft_consensus.cc:482] T b6467673a36449ce9d4d2aa649b75d4e P fd994f1fa3b2442f91a8df65847891f6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 02:13:04.830509 18941 raft_consensus.cc:3037] T b6467673a36449ce9d4d2aa649b75d4e P fd994f1fa3b2442f91a8df65847891f6 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:13:04.831481 18940 leader_election.cc:288] T b6467673a36449ce9d4d2aa649b75d4e P e17eecb4a4fe4e1aaee6b111171975b3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 222bec4e40b744999bced4a1584e7567 (127.6.235.65:38651), fd994f1fa3b2442f91a8df65847891f6 (127.6.235.67:34991)
I20260928 02:13:04.834308 18941 raft_consensus.cc:504] T b6467673a36449ce9d4d2aa649b75d4e P fd994f1fa3b2442f91a8df65847891f6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e17eecb4a4fe4e1aaee6b111171975b3" member_type: VOTER last_known_addr { host: "127.6.235.66" port: 43937 } } peers { permanent_uuid: "222bec4e40b744999bced4a1584e7567" member_type: VOTER last_known_addr { host: "127.6.235.65" port: 38651 } } peers { permanent_uuid: "fd994f1fa3b2442f91a8df65847891f6" member_type: VOTER last_known_addr { host: "127.6.235.67" port: 34991 } }
I20260928 02:13:04.835356 18735 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b6467673a36449ce9d4d2aa649b75d4e" candidate_uuid: "fd994f1fa3b2442f91a8df65847891f6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e17eecb4a4fe4e1aaee6b111171975b3"
I20260928 02:13:04.835593 18735 raft_consensus.cc:3037] T b6467673a36449ce9d4d2aa649b75d4e P e17eecb4a4fe4e1aaee6b111171975b3 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:13:04.835733 18592 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b6467673a36449ce9d4d2aa649b75d4e" candidate_uuid: "fd994f1fa3b2442f91a8df65847891f6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "222bec4e40b744999bced4a1584e7567"
I20260928 02:13:04.835938 18592 raft_consensus.cc:3037] T b6467673a36449ce9d4d2aa649b75d4e P 222bec4e40b744999bced4a1584e7567 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:13:04.838539 18592 raft_consensus.cc:2454] T b6467673a36449ce9d4d2aa649b75d4e P 222bec4e40b744999bced4a1584e7567 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fd994f1fa3b2442f91a8df65847891f6 in term 1.
I20260928 02:13:04.838847 18941 leader_election.cc:288] T b6467673a36449ce9d4d2aa649b75d4e P fd994f1fa3b2442f91a8df65847891f6 [CANDIDATE]: Term 1 election: Requested vote from peers e17eecb4a4fe4e1aaee6b111171975b3 (127.6.235.66:43937), 222bec4e40b744999bced4a1584e7567 (127.6.235.65:38651)
I20260928 02:13:04.838994 18735 raft_consensus.cc:2454] T b6467673a36449ce9d4d2aa649b75d4e P e17eecb4a4fe4e1aaee6b111171975b3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fd994f1fa3b2442f91a8df65847891f6 in term 1.
I20260928 02:13:04.839098 18805 leader_election.cc:302] T b6467673a36449ce9d4d2aa649b75d4e P fd994f1fa3b2442f91a8df65847891f6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 222bec4e40b744999bced4a1584e7567, fd994f1fa3b2442f91a8df65847891f6; no voters:
I20260928 02:13:04.839723 18941 raft_consensus.cc:2781] T b6467673a36449ce9d4d2aa649b75d4e P fd994f1fa3b2442f91a8df65847891f6 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:13:04.840251 18941 raft_consensus.cc:686] T b6467673a36449ce9d4d2aa649b75d4e P fd994f1fa3b2442f91a8df65847891f6 [term 1 LEADER]: Becoming Leader. State: Replica: fd994f1fa3b2442f91a8df65847891f6, State: Running, Role: LEADER
I20260928 02:13:04.840701 18930 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:13:04.832349 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.235.66:55161 (local address 127.6.235.65:38651)
0928 02:13:04.832598 (+ 249us) server_negotiation.cc:206] Beginning negotiation
0928 02:13:04.832603 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 02:13:04.832992 (+ 389us) server_negotiation.cc:407] Connection header received
0928 02:13:04.833237 (+ 245us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:13:04.833243 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:13:04.833321 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:13:04.833410 (+ 89us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:13:04.834190 (+ 780us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:13:04.835412 (+ 1222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:13:04.838276 (+ 2864us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:13:04.839567 (+ 1291us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:13:04.839750 (+ 183us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:13:04.839835 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0928 02:13:04.840283 (+ 448us) server_negotiation.cc:299] Negotiation successful
0928 02:13:04.840428 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":127}
I20260928 02:13:04.840808 18941 consensus_queue.cc:237] T b6467673a36449ce9d4d2aa649b75d4e P fd994f1fa3b2442f91a8df65847891f6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e17eecb4a4fe4e1aaee6b111171975b3" member_type: VOTER last_known_addr { host: "127.6.235.66" port: 43937 } } peers { permanent_uuid: "222bec4e40b744999bced4a1584e7567" member_type: VOTER last_known_addr { host: "127.6.235.65" port: 38651 } } peers { permanent_uuid: "fd994f1fa3b2442f91a8df65847891f6" member_type: VOTER last_known_addr { host: "127.6.235.67" port: 34991 } }
I20260928 02:13:04.841648 18943 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:13:04.831862 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.235.65:38651 (local address 127.6.235.66:55161)
0928 02:13:04.832855 (+ 993us) negotiation.cc:107] Waiting for socket to connect
0928 02:13:04.832885 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0928 02:13:04.833080 (+ 195us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:13:04.833580 (+ 500us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:13:04.833585 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:13:04.833610 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:13:04.833998 (+ 388us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:13:04.834014 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:13:04.835617 (+ 1603us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:13:04.835623 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:13:04.838058 (+ 2435us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:13:04.838074 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:13:04.839796 (+ 1722us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:13:04.839859 (+ 63us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:13:04.840042 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:13:04.840086 (+ 44us) client_negotiation.cc:769] Sending connection context
0928 02:13:04.841145 (+ 1059us) client_negotiation.cc:240] Negotiation successful
0928 02:13:04.841307 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":825,"thread_start_us":743,"threads_started":1}
I20260928 02:13:04.842415 18592 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b6467673a36449ce9d4d2aa649b75d4e" candidate_uuid: "e17eecb4a4fe4e1aaee6b111171975b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "222bec4e40b744999bced4a1584e7567" is_pre_election: true
I20260928 02:13:04.842752 18592 raft_consensus.cc:2379] T b6467673a36449ce9d4d2aa649b75d4e P 222bec4e40b744999bced4a1584e7567 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e17eecb4a4fe4e1aaee6b111171975b3 in current term 1: Already voted for candidate fd994f1fa3b2442f91a8df65847891f6 in this term.
I20260928 02:13:04.844383 18944 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:13:04.831909 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.235.67:34991 (local address 127.6.235.66:58659)
0928 02:13:04.833702 (+ 1793us) negotiation.cc:107] Waiting for socket to connect
0928 02:13:04.833718 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 02:13:04.833866 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:13:04.834352 (+ 486us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:13:04.834356 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:13:04.834379 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:13:04.834763 (+ 384us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:13:04.834775 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:13:04.839287 (+ 4512us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:13:04.839292 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:13:04.841790 (+ 2498us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:13:04.841807 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:13:04.843815 (+ 2008us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:13:04.843819 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:13:04.843932 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:13:04.843964 (+ 32us) client_negotiation.cc:769] Sending connection context
0928 02:13:04.844038 (+ 74us) client_negotiation.cc:240] Negotiation successful
0928 02:13:04.844139 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1664,"thread_start_us":1438,"threads_started":1}
I20260928 02:13:04.844683 18932 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:13:04.832013 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.235.66:58659 (local address 127.6.235.67:34991)
0928 02:13:04.832288 (+ 275us) server_negotiation.cc:206] Beginning negotiation
0928 02:13:04.832297 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 02:13:04.833808 (+ 1511us) server_negotiation.cc:407] Connection header received
0928 02:13:04.834032 (+ 224us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:13:04.834037 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:13:04.834111 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:13:04.834186 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:13:04.834942 (+ 756us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:13:04.836014 (+ 1072us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:13:04.842014 (+ 6000us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:13:04.843104 (+ 1090us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:13:04.843289 (+ 185us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:13:04.843370 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0928 02:13:04.844118 (+ 748us) server_negotiation.cc:299] Negotiation successful
0928 02:13:04.844331 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260928 02:13:04.845763 18869 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b6467673a36449ce9d4d2aa649b75d4e" candidate_uuid: "e17eecb4a4fe4e1aaee6b111171975b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fd994f1fa3b2442f91a8df65847891f6" is_pre_election: true
I20260928 02:13:04.846807 18670 leader_election.cc:302] T b6467673a36449ce9d4d2aa649b75d4e P e17eecb4a4fe4e1aaee6b111171975b3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e17eecb4a4fe4e1aaee6b111171975b3; no voters: 222bec4e40b744999bced4a1584e7567, fd994f1fa3b2442f91a8df65847891f6
I20260928 02:13:04.847245 18940 raft_consensus.cc:2726] T b6467673a36449ce9d4d2aa649b75d4e P e17eecb4a4fe4e1aaee6b111171975b3 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260928 02:13:04.848625 18423 catalog_manager.cc:5495] T b6467673a36449ce9d4d2aa649b75d4e P fd994f1fa3b2442f91a8df65847891f6 reported cstate change: term changed from 0 to 1, leader changed from <none> to fd994f1fa3b2442f91a8df65847891f6 (127.6.235.67). New cstate: current_term: 1 leader_uuid: "fd994f1fa3b2442f91a8df65847891f6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e17eecb4a4fe4e1aaee6b111171975b3" member_type: VOTER last_known_addr { host: "127.6.235.66" port: 43937 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "222bec4e40b744999bced4a1584e7567" member_type: VOTER last_known_addr { host: "127.6.235.65" port: 38651 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fd994f1fa3b2442f91a8df65847891f6" member_type: VOTER last_known_addr { host: "127.6.235.67" port: 34991 } health_report { overall_health: HEALTHY } } }
I20260928 02:13:04.871222 18932 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:13:04.865525 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39190 (local address 127.6.235.67:34991)
0928 02:13:04.865744 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0928 02:13:04.865750 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 02:13:04.865779 (+ 29us) server_negotiation.cc:407] Connection header received
0928 02:13:04.865990 (+ 211us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:13:04.865993 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:13:04.866082 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:13:04.866208 (+ 126us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 02:13:04.866622 (+ 414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:13:04.867394 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:13:04.868885 (+ 1491us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:13:04.869374 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:13:04.869480 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:13:04.869994 (+ 514us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 02:13:04.870560 (+ 566us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 02:13:04.870681 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0928 02:13:04.870784 (+ 103us) server_negotiation.cc:299] Negotiation successful
0928 02:13:04.870878 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260928 02:13:04.880086 18592 raft_consensus.cc:1261] T b6467673a36449ce9d4d2aa649b75d4e P 222bec4e40b744999bced4a1584e7567 [term 1 FOLLOWER]: Refusing update from remote peer fd994f1fa3b2442f91a8df65847891f6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 02:13:04.880142 18735 raft_consensus.cc:1261] T b6467673a36449ce9d4d2aa649b75d4e P e17eecb4a4fe4e1aaee6b111171975b3 [term 1 FOLLOWER]: Refusing update from remote peer fd994f1fa3b2442f91a8df65847891f6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 02:13:04.881206 18945 consensus_queue.cc:1035] T b6467673a36449ce9d4d2aa649b75d4e P fd994f1fa3b2442f91a8df65847891f6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e17eecb4a4fe4e1aaee6b111171975b3" member_type: VOTER last_known_addr { host: "127.6.235.66" port: 43937 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 02:13:04.881668 18941 consensus_queue.cc:1035] T b6467673a36449ce9d4d2aa649b75d4e P fd994f1fa3b2442f91a8df65847891f6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "222bec4e40b744999bced4a1584e7567" member_type: VOTER last_known_addr { host: "127.6.235.65" port: 38651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 02:13:04.901084 18949 mvcc.cc:205] Tried to move back new op lower bound from 7334140251657334784 to 7334140251512995840. Current Snapshot: MvccSnapshot[applied={T|T < 7334140251657334784 or (T in {7334140251657334784})}]
I20260928 02:13:04.913552 18424 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48296:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 02:13:04.913844 18424 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48296:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 02:13:04.917909 18424 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e09f2d99b867436286d683dc57e441fa: Sending DeleteTablet for 3 replicas of tablet b6467673a36449ce9d4d2aa649b75d4e
I20260928 02:13:04.919129 18572 tablet_service.cc:1514] Processing DeleteTablet for tablet b6467673a36449ce9d4d2aa649b75d4e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:13:04 UTC) from {username='kudu', principal='kudu/127.6.235.126@KRBTEST.COM'} at 127.0.0.1:59364
I20260928 02:13:04.919157 18715 tablet_service.cc:1514] Processing DeleteTablet for tablet b6467673a36449ce9d4d2aa649b75d4e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:13:04 UTC) from {username='kudu', principal='kudu/127.6.235.126@KRBTEST.COM'} at 127.0.0.1:59862
I20260928 02:13:04.920095 18959 tablet_replica.cc:330] T b6467673a36449ce9d4d2aa649b75d4e P 222bec4e40b744999bced4a1584e7567: stopping tablet replica
I20260928 02:13:04.920331 18849 tablet_service.cc:1514] Processing DeleteTablet for tablet b6467673a36449ce9d4d2aa649b75d4e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:13:04 UTC) from {username='kudu', principal='kudu/127.6.235.126@KRBTEST.COM'} at 127.0.0.1:39178
I20260928 02:13:04.920713 18959 raft_consensus.cc:2229] T b6467673a36449ce9d4d2aa649b75d4e P 222bec4e40b744999bced4a1584e7567 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 02:13:04.921334 18959 raft_consensus.cc:2258] T b6467673a36449ce9d4d2aa649b75d4e P 222bec4e40b744999bced4a1584e7567 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 02:13:04.921711 7085 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskTlaUuc/build/asan/bin/kudu pid 18504
I20260928 02:13:04.925050 18961 tablet_replica.cc:330] T b6467673a36449ce9d4d2aa649b75d4e P fd994f1fa3b2442f91a8df65847891f6: stopping tablet replica
I20260928 02:13:04.925601 18961 raft_consensus.cc:2229] T b6467673a36449ce9d4d2aa649b75d4e P fd994f1fa3b2442f91a8df65847891f6 [term 1 LEADER]: Raft consensus shutting down.
I20260928 02:13:04.926427 18961 raft_consensus.cc:2258] T b6467673a36449ce9d4d2aa649b75d4e P fd994f1fa3b2442f91a8df65847891f6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 02:13:04.928694 18961 ts_tablet_manager.cc:1905] T b6467673a36449ce9d4d2aa649b75d4e P fd994f1fa3b2442f91a8df65847891f6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 02:13:04.939337 18961 ts_tablet_manager.cc:1918] T b6467673a36449ce9d4d2aa649b75d4e P fd994f1fa3b2442f91a8df65847891f6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 02:13:04.939538 18961 log.cc:1196] T b6467673a36449ce9d4d2aa649b75d4e P fd994f1fa3b2442f91a8df65847891f6: Deleting WAL directory at /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790561432251112-7085-0/minicluster-data/ts-2/wal/wals/b6467673a36449ce9d4d2aa649b75d4e
I20260928 02:13:04.940110 18961 ts_tablet_manager.cc:1939] T b6467673a36449ce9d4d2aa649b75d4e P fd994f1fa3b2442f91a8df65847891f6: Deleting consensus metadata
I20260928 02:13:04.940940 18959 ts_tablet_manager.cc:1905] T b6467673a36449ce9d4d2aa649b75d4e P 222bec4e40b744999bced4a1584e7567: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 02:13:04.942039 18410 catalog_manager.cc:4841] TS fd994f1fa3b2442f91a8df65847891f6 (127.6.235.67:34991): tablet b6467673a36449ce9d4d2aa649b75d4e (table test-table [id=cfc4a9201edd43fa93b63e3717abbc86]) successfully deleted
I20260928 02:13:04.946650 18959 ts_tablet_manager.cc:1918] T b6467673a36449ce9d4d2aa649b75d4e P 222bec4e40b744999bced4a1584e7567: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 02:13:04.946839 18959 log.cc:1196] T b6467673a36449ce9d4d2aa649b75d4e P 222bec4e40b744999bced4a1584e7567: Deleting WAL directory at /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790561432251112-7085-0/minicluster-data/ts-0/wal/wals/b6467673a36449ce9d4d2aa649b75d4e
I20260928 02:13:04.947249 18959 ts_tablet_manager.cc:1939] T b6467673a36449ce9d4d2aa649b75d4e P 222bec4e40b744999bced4a1584e7567: Deleting consensus metadata
I20260928 02:13:04.947827 18960 tablet_replica.cc:330] T b6467673a36449ce9d4d2aa649b75d4e P e17eecb4a4fe4e1aaee6b111171975b3: stopping tablet replica
I20260928 02:13:04.948293 18960 raft_consensus.cc:2229] T b6467673a36449ce9d4d2aa649b75d4e P e17eecb4a4fe4e1aaee6b111171975b3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 02:13:04.948818 18960 raft_consensus.cc:2258] T b6467673a36449ce9d4d2aa649b75d4e P e17eecb4a4fe4e1aaee6b111171975b3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 02:13:04.948975 18411 catalog_manager.cc:4841] TS 222bec4e40b744999bced4a1584e7567 (127.6.235.65:38651): tablet b6467673a36449ce9d4d2aa649b75d4e (table test-table [id=cfc4a9201edd43fa93b63e3717abbc86]) successfully deleted
I20260928 02:13:04.956359 18960 ts_tablet_manager.cc:1905] T b6467673a36449ce9d4d2aa649b75d4e P e17eecb4a4fe4e1aaee6b111171975b3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 02:13:04.962998 18960 ts_tablet_manager.cc:1918] T b6467673a36449ce9d4d2aa649b75d4e P e17eecb4a4fe4e1aaee6b111171975b3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 02:13:04.963195 18960 log.cc:1196] T b6467673a36449ce9d4d2aa649b75d4e P e17eecb4a4fe4e1aaee6b111171975b3: Deleting WAL directory at /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790561432251112-7085-0/minicluster-data/ts-1/wal/wals/b6467673a36449ce9d4d2aa649b75d4e
I20260928 02:13:04.963653 18960 ts_tablet_manager.cc:1939] T b6467673a36449ce9d4d2aa649b75d4e P e17eecb4a4fe4e1aaee6b111171975b3: Deleting consensus metadata
I20260928 02:13:04.965307 18408 catalog_manager.cc:4841] TS e17eecb4a4fe4e1aaee6b111171975b3 (127.6.235.66:43937): tablet b6467673a36449ce9d4d2aa649b75d4e (table test-table [id=cfc4a9201edd43fa93b63e3717abbc86]) successfully deleted
Sep 28 02:13:04 dist-test-slave-86gv krb5kdc[17948](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561580, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.235.65@KRBTEST.COM
I20260928 02:13:04.978416 18930 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:13:04.922868 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59380 (local address 127.6.235.65:38651)
0928 02:13:04.923045 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0928 02:13:04.923052 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 02:13:04.953929 (+ 30877us) server_negotiation.cc:407] Connection header received
0928 02:13:04.954156 (+ 227us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:13:04.954163 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:13:04.954251 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:13:04.954363 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:13:04.956238 (+ 1875us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:13:04.957294 (+ 1056us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:13:04.959234 (+ 1940us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:13:04.959867 (+ 633us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:13:04.959970 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:13:04.971496 (+ 11526us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:13:04.971514 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:13:04.971536 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:13:04.971584 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:13:04.975153 (+ 3569us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:13:04.976097 (+ 944us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:13:04.976106 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:13:04.976109 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:13:04.976269 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:13:04.976701 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:13:04.976705 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:13:04.976707 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:13:04.977166 (+ 459us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:13:04.977486 (+ 320us) server_negotiation.cc:1035] Waiting for connection context
0928 02:13:04.977766 (+ 280us) server_negotiation.cc:299] Negotiation successful
0928 02:13:04.977946 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260928 02:13:04.979112 18632 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 02:13:05.109623 7085 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTlaUuc/build/asan/bin/kudu with pid 18504
I20260928 02:13:05.134999 7085 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskTlaUuc/build/asan/bin/kudu pid 18643
Sep 28 02:13:05 dist-test-slave-86gv krb5kdc[17948](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561580, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.235.66@KRBTEST.COM
I20260928 02:13:05.151831 18928 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:13:05.135789 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59864 (local address 127.6.235.66:43937)
0928 02:13:05.136019 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0928 02:13:05.136028 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 02:13:05.136051 (+ 23us) server_negotiation.cc:407] Connection header received
0928 02:13:05.136199 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:13:05.136205 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:13:05.136288 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:13:05.136401 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:13:05.137794 (+ 1393us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:13:05.138876 (+ 1082us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:13:05.140178 (+ 1302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:13:05.140794 (+ 616us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:13:05.141072 (+ 278us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:13:05.145671 (+ 4599us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:13:05.145686 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:13:05.145701 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:13:05.145739 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:13:05.148680 (+ 2941us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:13:05.149696 (+ 1016us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:13:05.149703 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:13:05.149706 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:13:05.149838 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:13:05.150265 (+ 427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:13:05.150268 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:13:05.150270 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:13:05.150673 (+ 403us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:13:05.150903 (+ 230us) server_negotiation.cc:1035] Waiting for connection context
0928 02:13:05.151316 (+ 413us) server_negotiation.cc:299] Negotiation successful
0928 02:13:05.151508 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260928 02:13:05.152498 18775 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 02:13:05.246660 7085 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTlaUuc/build/asan/bin/kudu with pid 18643
I20260928 02:13:05.270896 7085 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskTlaUuc/build/asan/bin/kudu pid 18786
Sep 28 02:13:05 dist-test-slave-86gv krb5kdc[17948](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561580, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.235.67@KRBTEST.COM
I20260928 02:13:05.290011 18932 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:13:05.271690 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39202 (local address 127.6.235.67:34991)
0928 02:13:05.271924 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0928 02:13:05.271930 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 02:13:05.271963 (+ 33us) server_negotiation.cc:407] Connection header received
0928 02:13:05.272127 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:13:05.272133 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:13:05.272228 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:13:05.272346 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:13:05.273699 (+ 1353us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:13:05.274829 (+ 1130us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:13:05.276454 (+ 1625us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:13:05.277164 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:13:05.277301 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:13:05.282410 (+ 5109us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:13:05.282426 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:13:05.282448 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:13:05.282502 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:13:05.286742 (+ 4240us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:13:05.287511 (+ 769us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:13:05.287527 (+ 16us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:13:05.287532 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:13:05.287697 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:13:05.288112 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:13:05.288118 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:13:05.288123 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:13:05.288685 (+ 562us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:13:05.289109 (+ 424us) server_negotiation.cc:1035] Waiting for connection context
0928 02:13:05.289364 (+ 255us) server_negotiation.cc:299] Negotiation successful
0928 02:13:05.289614 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":89}
I20260928 02:13:05.290874 18909 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 02:13:05.421479 7085 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTlaUuc/build/asan/bin/kudu with pid 18786
I20260928 02:13:05.442456 7085 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskTlaUuc/build/asan/bin/kudu pid 18392
I20260928 02:13:05.443682 18454 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 02:13:05.558495 7085 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTlaUuc/build/asan/bin/kudu with pid 18392
I20260928 02:13:05.580667 7085 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260928 02:13:05.916649 7085 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260928 02:13:05.916848 7085 mini_ranger.cc:66] Stopping Ranger...
I20260928 02:13:06.262861 7085 mini_ranger.cc:68] Stopped Ranger
2026-09-28 02:13:06.263 UTC [17978] LOG: received smart shutdown request
2026-09-28 02:13:06.267 UTC [17978] LOG: background worker "logical replication launcher" (PID 17987) exited with exit code 1
2026-09-28 02:13:06.268 UTC [17982] LOG: shutting down
2026-09-28 02:13:06.477 UTC [17978] LOG: database system is shut down
2026-09-28T02:13:06Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegration (51436 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers
Loading random data
Initializing database '/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790561432251112-7085-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 02:13:06 dist-test-slave-86gv krb5kdc[18975](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 02:13:06 dist-test-slave-86gv krb5kdc[18975](info): set up 2 sockets
Sep 28 02:13:06 dist-test-slave-86gv krb5kdc[18975](info): commencing operation
krb5kdc: starting...
W20260928 02:13:07.729693 7085 mini_kdc.cc:121] Time spent starting KDC: real 1.068s user 0.000s sys 0.025s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790561432251112-7085-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790561432251112-7085-0/krb5kdc/test-user.keytab.
Sep 28 02:13:07 dist-test-slave-86gv krb5kdc[18975](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561587, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T02:13:07Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T02:13:07Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.6.235.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.6.235.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.6.235.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790561432251112-7085-0/krb5kdc/rangeradmin_127.6.235.84@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.6.235.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790561432251112-7085-0/krb5kdc/rangeradmin_127.6.235.84@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.6.235.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.6.235.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.6.235.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790561432251112-7085-0/krb5kdc/rangerlookup_127.6.235.84@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.6.235.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790561432251112-7085-0/krb5kdc/rangerlookup_127.6.235.84@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.6.235.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.6.235.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.6.235.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790561432251112-7085-0/krb5kdc/HTTP_127.6.235.84@KRBTEST.COM.keytab.
Entry for principal HTTP/127.6.235.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790561432251112-7085-0/krb5kdc/HTTP_127.6.235.84@KRBTEST.COM.keytab.
I20260928 02:13:08.080987 7085 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790561432251112-7085-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskTlaUuc/build/asan/bin/postgres/pg_ctl -D /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790561432251112-7085-0/minicluster-data/postgres -l logfile start
2026-09-28 02:13:10.811 UTC [19005] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-28 02:13:10.812 UTC [19005] LOG: listening on IPv4 address "127.6.235.84", port 51587
2026-09-28 02:13:10.815 UTC [19005] LOG: listening on Unix socket "/tmp/.s.PGSQL.51587"
2026-09-28 02:13:10.839 UTC [19008] LOG: database system was shut down at 2026-09-28 02:13:09 UTC
2026-09-28 02:13:10.843 UTC [19005] LOG: database system is ready to accept connections
I20260928 02:13:11.821632 7085 mini_postgres.cc:96] Postgres bound to 51587
2026-09-28 02:13:11.840 UTC [19016] FATAL: database "slave" does not exist
127.6.235.84:51587 - accepting connections
I20260928 02:13:11.841650 7085 mini_ranger.cc:161] Starting Ranger...
I20260928 02:13:11.871059 7085 mini_ranger.cc:84] Created miniranger Postgres user
I20260928 02:13:12.458796 7085 mini_ranger.cc:87] Created ranger Postgres database
I20260928 02:13:12.458922 7085 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790561432251112-7085-0/minicluster-data/ranger-admin
2026-09-28 02:13:12,756 [I] DB FLAVOR :POSTGRES
2026-09-28 02:13:12,756 [I] --------- Verifying Ranger DB connection ---------
2026-09-28 02:13:12,756 [I] Checking connection..
2026-09-28 02:13:12,757 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskTlaUuc/build/asan/bin/postgresql.jar:/tmp/dist-test-taskTlaUuc/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.235.84:51587/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-28 02:13:13,023 [I] Checking connection passed.
2026-09-28 02:13:13,023 [I] --------- Verifying version history table ---------
2026-09-28 02:13:13,023 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskTlaUuc/build/asan/bin/postgresql.jar:/tmp/dist-test-taskTlaUuc/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.235.84:51587/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-28 02:13:13,261 [I] Table x_db_version_h does not exist in database ranger
2026-09-28 02:13:13,262 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskTlaUuc/build/asan/bin/postgresql.jar:/tmp/dist-test-taskTlaUuc/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.235.84:51587/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-28 02:13:13,505 [I] Table x_db_version_h does not exist in database ranger
2026-09-28 02:13:13,505 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-28 02:13:13,505 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskTlaUuc/build/asan/bin/postgresql.jar:/tmp/dist-test-taskTlaUuc/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.235.84:51587/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskTlaUuc/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-28 02:13:13.775 UTC [19080] WARNING: there is no transaction in progress
2026-09-28 02:13:13,782 [I] create_dbversion_catalog.sql file imported successfully
2026-09-28 02:13:13,782 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskTlaUuc/build/asan/bin/postgresql.jar:/tmp/dist-test-taskTlaUuc/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.235.84:51587/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-28 02:13:14,044 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-28 02:13:14,045 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-28 02:13:14,045 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskTlaUuc/build/asan/bin/postgresql.jar:/tmp/dist-test-taskTlaUuc/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.235.84:51587/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-28 02:13:14,311 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskTlaUuc/build/asan/bin/postgresql.jar:/tmp/dist-test-taskTlaUuc/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.235.84:51587/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-28 02:13:14,565 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskTlaUuc/build/asan/bin/postgresql.jar:/tmp/dist-test-taskTlaUuc/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.235.84:51587/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-86gv.c.gcp-upstream.internal','N') ;"
2026-09-28 02:13:14,806 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskTlaUuc/build/asan/bin/postgresql.jar:/tmp/dist-test-taskTlaUuc/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.235.84:51587/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-28 02:13:15,065 [I] Table x_portal_user does not exist in database ranger
2026-09-28 02:13:15,065 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskTlaUuc/build/asan/bin/postgresql.jar:/tmp/dist-test-taskTlaUuc/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.235.84:51587/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-28 02:13:15,307 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-28 02:13:15,307 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskTlaUuc/build/asan/bin/postgresql.jar:/tmp/dist-test-taskTlaUuc/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.235.84:51587/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-28 02:13:15,543 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskTlaUuc/build/asan/bin/postgresql.jar:/tmp/dist-test-taskTlaUuc/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.235.84:51587/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-28 02:13:15,784 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-28 02:13:15,785 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskTlaUuc/build/asan/bin/postgresql.jar:/tmp/dist-test-taskTlaUuc/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.235.84:51587/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskTlaUuc/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-28 02:13:16.841 UTC [19188] WARNING: there is no transaction in progress
2026-09-28 02:13:16.859 UTC [19188] WARNING: there is no transaction in progress
2026-09-28 02:13:16.875 UTC [19188] WARNING: there is no transaction in progress
2026-09-28 02:13:16.892 UTC [19188] WARNING: there is no transaction in progress
2026-09-28 02:13:16.909 UTC [19188] WARNING: there is no transaction in progress
2026-09-28 02:13:16.928 UTC [19188] WARNING: there is no transaction in progress
2026-09-28 02:13:16.987 UTC [19188] WARNING: there is no transaction in progress
2026-09-28 02:13:16.993 UTC [19188] WARNING: there is no transaction in progress
2026-09-28 02:13:17.009 UTC [19188] WARNING: there is no transaction in progress
2026-09-28 02:13:17.019 UTC [19188] WARNING: there is no transaction in progress
2026-09-28 02:13:17.030 UTC [19188] WARNING: there is no transaction in progress
2026-09-28 02:13:17.044 UTC [19188] WARNING: there is no transaction in progress
2026-09-28 02:13:17.054 UTC [19188] WARNING: there is no transaction in progress
2026-09-28 02:13:17.061 UTC [19188] WARNING: there is no transaction in progress
2026-09-28 02:13:17.069 UTC [19188] WARNING: there is no transaction in progress
2026-09-28 02:13:17.076 UTC [19188] WARNING: there is no transaction in progress
2026-09-28 02:13:17,749 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-28 02:13:17,749 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskTlaUuc/build/asan/bin/postgresql.jar:/tmp/dist-test-taskTlaUuc/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.235.84:51587/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-28 02:13:18,005 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-28 02:13:18,005 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskTlaUuc/build/asan/bin/postgresql.jar:/tmp/dist-test-taskTlaUuc/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.235.84:51587/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-28 02:13:18,252 [I] Table x_portal_user already exists in database 'ranger'
2026-09-28 02:13:18,252 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskTlaUuc/build/asan/bin/postgresql.jar:/tmp/dist-test-taskTlaUuc/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.235.84:51587/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-28 02:13:18,515 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-28 02:13:18,516 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskTlaUuc/build/asan/bin/postgresql.jar:/tmp/dist-test-taskTlaUuc/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.235.84:51587/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-28 02:13:18,770 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskTlaUuc/build/asan/bin/postgresql.jar:/tmp/dist-test-taskTlaUuc/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.235.84:51587/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-28 02:13:19,014 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskTlaUuc/build/asan/bin/postgresql.jar:/tmp/dist-test-taskTlaUuc/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.235.84:51587/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-86gv.c.gcp-upstream.internal';"
2026-09-28 02:13:19,237 [I] CORE_DB_SCHEMA import status has been updated
2026-09-28 02:13:19,237 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskTlaUuc/build/asan/bin/postgresql.jar:/tmp/dist-test-taskTlaUuc/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.235.84:51587/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-28 02:13:19,473 [I] DB_PATCHES have already been applied
I20260928 02:13:19.479219 7085 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790561432251112-7085-0/minicluster-data/ranger-admin:/tmp/dist-test-taskTlaUuc/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskTlaUuc/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskTlaUuc/build/asan/bin/postgresql.jar:/tmp/dist-test-taskTlaUuc/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260928 02:13:19.479307 7085 mini_ranger.cc:193] Using host: 127.6.235.84
I20260928 02:13:19.495070 7085 mini_ranger.cc:227] Ranger admin URL: http://127.6.235.84:49131
Sep 28, 2026 2:13:19 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 28, 2026 2:13:19 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 28, 2026 2:13:19 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790561432251112-7085-0/minicluster-data/ranger-admin/ews/webapp
Sep 28, 2026 2:13:19 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790561432251112-7085-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 28, 2026 2:13:19 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790561432251112-7085-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 28, 2026 2:13:20 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790561432251112-7085-0/minicluster-data/ranger-admin/ews/webapp].
Sep 28, 2026 2:13:20 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260928 02:13:20.639227 7085 mini_ranger.cc:160] Time spent starting Ranger: real 8.798s user 0.000s sys 0.043s
Sep 28, 2026 2:13:41 AM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 28, 2026 2:13:41 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.UserREST
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.MetricsREST
Sep 28, 2026 2:13:41 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 28, 2026 2:13:41 AM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 28, 2026 2:13:41 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 28, 2026 2:13:41 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 28, 2026 2:13:41 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 28, 2026 2:13:41 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 28, 2026 2:13:41 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 28, 2026 2:13:41 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 28, 2026 2:13:41 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 28, 2026 2:13:41 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 28, 2026 2:13:41 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 28, 2026 2:13:41 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 28, 2026 2:13:41 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 28, 2026 2:13:41 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 28, 2026 2:13:41 AM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 28, 2026 2:13:42 AM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260928 02:13:43.329629 7085 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.6.235.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.6.235.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.6.235.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790561432251112-7085-0/krb5kdc/rangerkms_127.6.235.84@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.6.235.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790561432251112-7085-0/krb5kdc/rangerkms_127.6.235.84@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.6.235.84@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.6.235.84@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.6.235.84@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790561432251112-7085-0/krb5kdc/HTTP_127.6.235.84@KRBTEST.COM.keytab.
Entry for principal HTTP/127.6.235.84@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790561432251112-7085-0/krb5kdc/HTTP_127.6.235.84@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 28 02:13:43 dist-test-slave-86gv krb5kdc[18975](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561623, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260928 02:13:43.514340 7085 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260928 02:13:43.536803 7085 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260928 02:13:44.536314 7085 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260928 02:13:44.536453 7085 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790561432251112-7085-0/minicluster-data/ranger-kms
I20260928 02:13:44.536499 7085 mini_ranger_kms.cc:227] Using postgres at 127.6.235.84:51587
2026-09-28 02:13:44,913 [I] DB FLAVOR :POSTGRES
2026-09-28 02:13:44,913 [I] --------- Verifying Ranger DB connection ---------
2026-09-28 02:13:44,913 [I] Checking connection
2026-09-28 02:13:44,913 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskTlaUuc/build/asan/bin/postgresql.jar:/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790561432251112-7085-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.235.84:51587/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-28 02:13:45,153 [I] connection success
2026-09-28 02:13:45,153 [I] --------- Verifying Ranger DB tables ---------
2026-09-28 02:13:45,153 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-28 02:13:45,153 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskTlaUuc/build/asan/bin/postgresql.jar:/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790561432251112-7085-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.235.84:51587/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-28 02:13:45,409 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-28 02:13:45,409 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-28 02:13:45,409 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-28 02:13:45,409 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskTlaUuc/build/asan/bin/postgresql.jar:/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790561432251112-7085-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.235.84:51587/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790561432251112-7085-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-28 02:13:45,695 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260928 02:13:46.209388 7085 mini_ranger_kms.cc:310] Created kms service
I20260928 02:13:46.445340 7085 mini_ranger_kms.cc:326] Created kudu user
I20260928 02:13:46.493814 7085 mini_ranger_kms.cc:343] Created rangerkms user
Sep 28 02:13:46 dist-test-slave-86gv krb5kdc[18975](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561623, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.6.235.84@KRBTEST.COM
I20260928 02:13:46.918391 7085 mini_ranger_kms.cc:384] Added ranger policy
I20260928 02:13:46.918722 7085 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790561432251112-7085-0/minicluster-data/ranger-kms:/tmp/dist-test-taskTlaUuc/build/asan/bin/postgresql.jar:/tmp/dist-test-taskTlaUuc/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskTlaUuc/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260928 02:13:46.918777 7085 mini_ranger_kms.cc:240] Using host: 127.6.235.84
I20260928 02:13:46.938597 7085 mini_ranger_kms.cc:276] Ranger KMS PID: 19369
I20260928 02:13:46.938753 7085 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.6.235.84:35091
Sep 28, 2026 2:13:47 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 28, 2026 2:13:47 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 28, 2026 2:13:48 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790561432251112-7085-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 28, 2026 2:13:48 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790561432251112-7085-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 28, 2026 2:13:48 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790561432251112-7085-0/minicluster-data/ranger-kms/ews/webapp].
Sep 28, 2026 2:13:48 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
Sep 28 02:13:49 dist-test-slave-86gv krb5kdc[18975](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790561629, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.6.235.84@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:13:49.290285 7085 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.776s user 0.076s sys 0.316s
I20260928 02:13:49.290548 7085 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260928 02:13:49.290742 7085 mini_ranger_kms.cc:401] 127.6.235.84:35091/kms/v1/keys
Sep 28 02:13:49 dist-test-slave-86gv krb5kdc[18975](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561629, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.6.235.84@KRBTEST.COM for HTTP/127.6.235.84@KRBTEST.COM
Sep 28 02:13:49 dist-test-slave-86gv krb5kdc[18975](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561629, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.6.235.84@KRBTEST.COM for HTTP/127.6.235.84@KRBTEST.COM
Sep 28 02:13:52 dist-test-slave-86gv krb5kdc[18975](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561632, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.6.235.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.235.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.235.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790561432251112-7085-0/krb5kdc/kudu_127.6.235.126.keytab.
Entry for principal kudu/127.6.235.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790561432251112-7085-0/krb5kdc/kudu_127.6.235.126.keytab.
I20260928 02:13:52.245309 7085 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTlaUuc/build/asan/bin/kudu
/tmp/dist-test-taskTlaUuc/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790561432251112-7085-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790561432251112-7085-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790561432251112-7085-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790561432251112-7085-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.235.126:41799
--webserver_interface=127.6.235.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790561432251112-7085-0/krb5kdc/kudu_127.6.235.126.keytab
--principal=kudu/127.6.235.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.235.84:40767
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.235.126:41799
--ranger_config_path=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790561432251112-7085-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.6.235.84:35091/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790561432251112-7085-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790561432251112-7085-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790561432251112-7085-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:13:52.600590 19419 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:13:52.601013 19419 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:13:52.601188 19419 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:13:52.608413 19419 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 02:13:52.608592 19419 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:13:52.608711 19419 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:13:52.608821 19419 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 02:13:52.608913 19419 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 02:13:52.619223 19419 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.235.84:40767
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790561432251112-7085-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790561432251112-7085-0/minicluster-data/master-0/wal
--ranger_kms_url=127.6.235.84:35091/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.6.235.126:41799
--ranger_config_path=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790561432251112-7085-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790561432251112-7085-0/krb5kdc/kudu_127.6.235.126.keytab
--principal=kudu/127.6.235.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.235.126:41799
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790561432251112-7085-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.235.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790561432251112-7085-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 00:51:59 UTC on ccycloud.quasar-axgkwn.root.comops.site
build id 4676464
ASAN enabled
I20260928 02:13:52.620730 19419 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:13:52.622131 19419 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:13:52.636324 19424 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:13:52.636343 19425 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:13:52.636617 19419 server_base.cc:1034] running on GCE node
W20260928 02:13:52.636385 19427 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:13:52.637754 19419 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:13:52.639353 19419 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:13:52.640609 19419 hybrid_clock.cc:648] HybridClock initialized: now 1790561632640544 us; error 42 us; skew 500 ppm
I20260928 02:13:52.641088 19419 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 02:13:52 dist-test-slave-86gv krb5kdc[18975](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561632, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.235.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:13:52.644683 19419 init.cc:377] Logged in from keytab as kudu/127.6.235.126@KRBTEST.COM (short username kudu)
I20260928 02:13:52.646924 19419 webserver.cc:460] Webserver started at http://127.6.235.126:41575/ using document root <none> and password file <none>
I20260928 02:13:52.647922 19419 fs_manager.cc:359] Metadata directory not provided
I20260928 02:13:52.648061 19419 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:13:52.648418 19419 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 28 02:13:52 dist-test-slave-86gv krb5kdc[18975](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561632, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.235.126@KRBTEST.COM for HTTP/127.6.235.84@KRBTEST.COM
I20260928 02:13:52.708678 19419 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790561432251112-7085-0/minicluster-data/master-0/data/instance:
uuid: "5948defd0e5e4ca39ca7d4f2883db7eb"
format_stamp: "Formatted at 2026-09-28 02:13:52 on dist-test-slave-86gv"
server_key: "9db666a31f29b0944d608b8a612f51e6"
server_key_iv: "f0b95912fc1bb4991cd3fb265bba55df"
server_key_version: "kuduclusterkey@0"
I20260928 02:13:52.709913 19419 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790561432251112-7085-0/minicluster-data/master-0/wal/instance:
uuid: "5948defd0e5e4ca39ca7d4f2883db7eb"
format_stamp: "Formatted at 2026-09-28 02:13:52 on dist-test-slave-86gv"
server_key: "9db666a31f29b0944d608b8a612f51e6"
server_key_iv: "f0b95912fc1bb4991cd3fb265bba55df"
server_key_version: "kuduclusterkey@0"
I20260928 02:13:52.717046 19419 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.006s sys 0.002s
I20260928 02:13:52.733866 19434 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:13:52.735555 19419 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260928 02:13:52.735780 19419 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790561432251112-7085-0/minicluster-data/master-0/data,/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790561432251112-7085-0/minicluster-data/master-0/wal
uuid: "5948defd0e5e4ca39ca7d4f2883db7eb"
format_stamp: "Formatted at 2026-09-28 02:13:52 on dist-test-slave-86gv"
server_key: "9db666a31f29b0944d608b8a612f51e6"
server_key_iv: "f0b95912fc1bb4991cd3fb265bba55df"
server_key_version: "kuduclusterkey@0"
I20260928 02:13:52.736034 19419 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790561432251112-7085-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790561432251112-7085-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790561432251112-7085-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:13:52.769464 19419 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:13:52.773556 19419 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:13:52.773897 19419 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:13:52.792300 19419 rpc_server.cc:307] RPC server started. Bound to: 127.6.235.126:41799
I20260928 02:13:52.792325 19485 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.235.126:41799 every 8 connection(s)
I20260928 02:13:52.793817 19419 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790561432251112-7085-0/minicluster-data/master-0/data/info.pb
I20260928 02:13:52.797716 19486 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:13:52.803969 7085 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTlaUuc/build/asan/bin/kudu as pid 19419
I20260928 02:13:52.804196 7085 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790561432251112-7085-0/minicluster-data/master-0/wal/instance
I20260928 02:13:52.804849 7085 external_mini_cluster.cc:1428] Setting key b79c4c8935039abe674aa1a04b057bcc
I20260928 02:13:52.809049 19486 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5948defd0e5e4ca39ca7d4f2883db7eb: Bootstrap starting.
I20260928 02:13:52.812361 19486 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5948defd0e5e4ca39ca7d4f2883db7eb: Neither blocks nor log segments found. Creating new log.
I20260928 02:13:52.813539 19486 log.cc:824] T 00000000000000000000000000000000 P 5948defd0e5e4ca39ca7d4f2883db7eb: Log is configured to *not* fsync() on all Append() calls
I20260928 02:13:52.816581 19486 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5948defd0e5e4ca39ca7d4f2883db7eb: No bootstrap required, opened a new log
Sep 28 02:13:52 dist-test-slave-86gv krb5kdc[18975](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561632, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.235.126@KRBTEST.COM
I20260928 02:13:52.825282 19486 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5948defd0e5e4ca39ca7d4f2883db7eb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5948defd0e5e4ca39ca7d4f2883db7eb" member_type: VOTER last_known_addr { host: "127.6.235.126" port: 41799 } }
I20260928 02:13:52.825700 19486 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5948defd0e5e4ca39ca7d4f2883db7eb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:13:52.825832 19486 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5948defd0e5e4ca39ca7d4f2883db7eb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5948defd0e5e4ca39ca7d4f2883db7eb, State: Initialized, Role: FOLLOWER
I20260928 02:13:52.826547 19486 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5948defd0e5e4ca39ca7d4f2883db7eb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5948defd0e5e4ca39ca7d4f2883db7eb" member_type: VOTER last_known_addr { host: "127.6.235.126" port: 41799 } }
I20260928 02:13:52.826857 19486 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5948defd0e5e4ca39ca7d4f2883db7eb [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 02:13:52.827005 19486 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5948defd0e5e4ca39ca7d4f2883db7eb [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 02:13:52.827222 19486 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5948defd0e5e4ca39ca7d4f2883db7eb [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:13:52.828939 19489 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:13:52.807125 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54520 (local address 127.6.235.126:41799)
0928 02:13:52.807927 (+ 802us) server_negotiation.cc:206] Beginning negotiation
0928 02:13:52.807940 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0928 02:13:52.807984 (+ 44us) server_negotiation.cc:407] Connection header received
0928 02:13:52.809862 (+ 1878us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:13:52.809888 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:13:52.810308 (+ 420us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:13:52.810744 (+ 436us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:13:52.812173 (+ 1429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:13:52.813612 (+ 1439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:13:52.815430 (+ 1818us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:13:52.816139 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:13:52.816344 (+ 205us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:13:52.820471 (+ 4127us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:13:52.820479 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:13:52.820496 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:13:52.820534 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:13:52.823857 (+ 3323us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:13:52.824749 (+ 892us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:13:52.824760 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:13:52.824774 (+ 14us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:13:52.825017 (+ 243us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:13:52.825414 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:13:52.825420 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:13:52.825423 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:13:52.826229 (+ 806us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:13:52.826485 (+ 256us) server_negotiation.cc:1035] Waiting for connection context
0928 02:13:52.827159 (+ 674us) server_negotiation.cc:299] Negotiation successful
0928 02:13:52.827638 (+ 479us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":589,"thread_start_us":364,"threads_started":1}
I20260928 02:13:52.830487 19486 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5948defd0e5e4ca39ca7d4f2883db7eb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5948defd0e5e4ca39ca7d4f2883db7eb" member_type: VOTER last_known_addr { host: "127.6.235.126" port: 41799 } }
I20260928 02:13:52.831097 19486 leader_election.cc:302] T 00000000000000000000000000000000 P 5948defd0e5e4ca39ca7d4f2883db7eb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5948defd0e5e4ca39ca7d4f2883db7eb; no voters:
I20260928 02:13:52.831782 19486 leader_election.cc:288] T 00000000000000000000000000000000 P 5948defd0e5e4ca39ca7d4f2883db7eb [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 02:13:52.831933 19491 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5948defd0e5e4ca39ca7d4f2883db7eb [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:13:52.833724 19491 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5948defd0e5e4ca39ca7d4f2883db7eb [term 1 LEADER]: Becoming Leader. State: Replica: 5948defd0e5e4ca39ca7d4f2883db7eb, State: Running, Role: LEADER
I20260928 02:13:52.834187 19486 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5948defd0e5e4ca39ca7d4f2883db7eb [sys.catalog]: configured and running, proceeding with master startup.
I20260928 02:13:52.834236 19491 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5948defd0e5e4ca39ca7d4f2883db7eb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5948defd0e5e4ca39ca7d4f2883db7eb" member_type: VOTER last_known_addr { host: "127.6.235.126" port: 41799 } }
I20260928 02:13:52.837200 19493 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5948defd0e5e4ca39ca7d4f2883db7eb [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5948defd0e5e4ca39ca7d4f2883db7eb. Latest consensus state: current_term: 1 leader_uuid: "5948defd0e5e4ca39ca7d4f2883db7eb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5948defd0e5e4ca39ca7d4f2883db7eb" member_type: VOTER last_known_addr { host: "127.6.235.126" port: 41799 } } }
I20260928 02:13:52.837625 19493 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5948defd0e5e4ca39ca7d4f2883db7eb [sys.catalog]: This master's current role is: LEADER
I20260928 02:13:52.838510 19500 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 02:13:52.838567 19492 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5948defd0e5e4ca39ca7d4f2883db7eb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5948defd0e5e4ca39ca7d4f2883db7eb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5948defd0e5e4ca39ca7d4f2883db7eb" member_type: VOTER last_known_addr { host: "127.6.235.126" port: 41799 } } }
I20260928 02:13:52.838846 19492 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5948defd0e5e4ca39ca7d4f2883db7eb [sys.catalog]: This master's current role is: LEADER
I20260928 02:13:52.845547 19500 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 02:13:52.846357 19486 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790561432251112-7085-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260928 02:13:52.865756 19500 catalog_manager.cc:1347] Generated new cluster ID: 58fbf9e1b9bb455ca74c27c45b660094
I20260928 02:13:52.865929 19500 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 02:13:52.897810 19500 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 02:13:52.899842 19500 catalog_manager.cc:1504] Loading token signing keys...
I20260928 02:13:52.927385 19500 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5948defd0e5e4ca39ca7d4f2883db7eb: Generated new TSK 0
I20260928 02:13:52.928580 19500 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.6.235.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.235.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.235.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790561432251112-7085-0/krb5kdc/kudu_127.6.235.65.keytab.
Entry for principal kudu/127.6.235.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790561432251112-7085-0/krb5kdc/kudu_127.6.235.65.keytab.
I20260928 02:13:53.669358 7085 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTlaUuc/build/asan/bin/kudu
/tmp/dist-test-taskTlaUuc/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790561432251112-7085-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790561432251112-7085-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790561432251112-7085-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790561432251112-7085-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.6.235.84:35091/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.235.65:0
--local_ip_for_outbound_sockets=127.6.235.65
--webserver_interface=127.6.235.65
--webserver_port=0
--tserver_master_addrs=127.6.235.126:41799
--keytab_file=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790561432251112-7085-0/krb5kdc/kudu_127.6.235.65.keytab
--principal=kudu/127.6.235.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.235.84:40767
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790561432251112-7085-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790561432251112-7085-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790561432251112-7085-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 28 02:13:53 dist-test-slave-86gv krb5kdc[18975](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790561633, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.235.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260928 02:13:54.119350 19531 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:13:54.119856 19531 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:13:54.120074 19531 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:13:54.132441 19531 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:13:54.132596 19531 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:13:54.132825 19531 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.235.65
I20260928 02:13:54.149006 19531 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.235.84:40767
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790561432251112-7085-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790561432251112-7085-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.6.235.84:35091/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790561432251112-7085-0/krb5kdc/kudu_127.6.235.65.keytab
--principal=kudu/127.6.235.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.235.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790561432251112-7085-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.6.235.65
--webserver_port=0
--tserver_master_addrs=127.6.235.126:41799
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.235.65
--log_dir=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790561432251112-7085-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 00:51:59 UTC on ccycloud.quasar-axgkwn.root.comops.site
build id 4676464
ASAN enabled
I20260928 02:13:54.150800 19531 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:13:54.152563 19531 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:13:54.195673 19541 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:13:54.209868 19542 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:13:54.212175 19545 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:13:54.213124 19531 server_base.cc:1034] running on GCE node
I20260928 02:13:54.214123 19531 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:13:54.220283 19531 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:13:54.221750 19531 hybrid_clock.cc:648] HybridClock initialized: now 1790561634221633 us; error 118 us; skew 500 ppm
I20260928 02:13:54.222216 19531 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 02:13:54 dist-test-slave-86gv krb5kdc[18975](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561634, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.235.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:13:54.226612 19531 init.cc:377] Logged in from keytab as kudu/127.6.235.65@KRBTEST.COM (short username kudu)
I20260928 02:13:54.232771 19531 webserver.cc:460] Webserver started at http://127.6.235.65:40937/ using document root <none> and password file <none>
I20260928 02:13:54.233870 19531 fs_manager.cc:359] Metadata directory not provided
I20260928 02:13:54.234082 19531 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:13:54.234551 19531 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 28 02:13:54 dist-test-slave-86gv krb5kdc[18975](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561634, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.235.65@KRBTEST.COM for HTTP/127.6.235.84@KRBTEST.COM
I20260928 02:13:54.257869 19531 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790561432251112-7085-0/minicluster-data/ts-0/data/instance:
uuid: "269968aeb430404a8977927db1ba1660"
format_stamp: "Formatted at 2026-09-28 02:13:54 on dist-test-slave-86gv"
server_key: "1920d280ef95171b6b956cd0b5f379d9"
server_key_iv: "efa66e220e9b8791b81d9266138266b8"
server_key_version: "kuduclusterkey@0"
I20260928 02:13:54.259048 19531 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790561432251112-7085-0/minicluster-data/ts-0/wal/instance:
uuid: "269968aeb430404a8977927db1ba1660"
format_stamp: "Formatted at 2026-09-28 02:13:54 on dist-test-slave-86gv"
server_key: "1920d280ef95171b6b956cd0b5f379d9"
server_key_iv: "efa66e220e9b8791b81d9266138266b8"
server_key_version: "kuduclusterkey@0"
I20260928 02:13:54.292985 19531 fs_manager.cc:691] Time spent creating directory manager: real 0.033s user 0.021s sys 0.012s
I20260928 02:13:54.325757 19551 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:13:54.327771 19531 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260928 02:13:54.327976 19531 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790561432251112-7085-0/minicluster-data/ts-0/data,/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790561432251112-7085-0/minicluster-data/ts-0/wal
uuid: "269968aeb430404a8977927db1ba1660"
format_stamp: "Formatted at 2026-09-28 02:13:54 on dist-test-slave-86gv"
server_key: "1920d280ef95171b6b956cd0b5f379d9"
server_key_iv: "efa66e220e9b8791b81d9266138266b8"
server_key_version: "kuduclusterkey@0"
I20260928 02:13:54.328245 19531 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790561432251112-7085-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790561432251112-7085-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790561432251112-7085-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:13:54.384037 19531 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:13:54.389856 19531 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:13:54.390219 19531 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:13:54.394274 19531 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:13:54.396296 19531 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:13:54.396399 19531 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:13:54.396534 19531 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:13:54.396595 19531 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 28 02:13:54 dist-test-slave-86gv krb5kdc[18975](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561633, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.235.126@KRBTEST.COM for HTTP/127.6.235.84@KRBTEST.COM
I20260928 02:13:54.513687 19531 rpc_server.cc:307] RPC server started. Bound to: 127.6.235.65:36397
I20260928 02:13:54.513713 19663 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.235.65:36397 every 8 connection(s)
I20260928 02:13:54.515398 19531 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790561432251112-7085-0/minicluster-data/ts-0/data/info.pb
I20260928 02:13:54.517378 7085 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTlaUuc/build/asan/bin/kudu as pid 19531
I20260928 02:13:54.517598 7085 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790561432251112-7085-0/minicluster-data/ts-0/wal/instance
I20260928 02:13:54.518404 7085 external_mini_cluster.cc:1428] Setting key 330af8aac5bf3d3141bf46fa9fd953f3
Sep 28 02:13:54 dist-test-slave-86gv krb5kdc[18975](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561634, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.235.65@KRBTEST.COM for kudu/127.6.235.126@KRBTEST.COM
I20260928 02:13:54.551486 19667 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:13:54.522113 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.235.126:41799 (local address 127.6.235.65:37479)
0928 02:13:54.523753 (+ 1640us) negotiation.cc:107] Waiting for socket to connect
0928 02:13:54.523810 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0928 02:13:54.526315 (+ 2505us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:13:54.530561 (+ 4246us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:13:54.530577 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:13:54.531443 (+ 866us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:13:54.532406 (+ 963us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:13:54.532437 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:13:54.534151 (+ 1714us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:13:54.534158 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:13:54.536506 (+ 2348us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:13:54.536525 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:13:54.537954 (+ 1429us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:13:54.537961 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:13:54.538280 (+ 319us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:13:54.539149 (+ 869us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:13:54.539179 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:13:54.541989 (+ 2810us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:13:54.546351 (+ 4362us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:13:54.546361 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:13:54.546382 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:13:54.547064 (+ 682us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:13:54.547547 (+ 483us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:13:54.547552 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:13:54.547555 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:13:54.547801 (+ 246us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:13:54.548539 (+ 738us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:13:54.548548 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:13:54.549068 (+ 520us) client_negotiation.cc:769] Sending connection context
0928 02:13:54.549381 (+ 313us) client_negotiation.cc:240] Negotiation successful
0928 02:13:54.549973 (+ 592us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1314,"thread_start_us":8073,"threads_started":1}
I20260928 02:13:54.552539 19666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:13:54.521162 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.235.65:37479 (local address 127.6.235.126:41799)
0928 02:13:54.529357 (+ 8195us) server_negotiation.cc:206] Beginning negotiation
0928 02:13:54.529665 (+ 308us) server_negotiation.cc:399] Waiting for connection header
0928 02:13:54.529696 (+ 31us) server_negotiation.cc:407] Connection header received
0928 02:13:54.529770 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:13:54.529775 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:13:54.529869 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:13:54.530011 (+ 142us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:13:54.532654 (+ 2643us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:13:54.533929 (+ 1275us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:13:54.536730 (+ 2801us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:13:54.537452 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:13:54.537589 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:13:54.542297 (+ 4708us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:13:54.542304 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:13:54.542311 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:13:54.542353 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:13:54.546131 (+ 3778us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:13:54.547227 (+ 1096us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:13:54.547233 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:13:54.547237 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:13:54.547399 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:13:54.547954 (+ 555us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:13:54.547959 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:13:54.547962 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:13:54.548317 (+ 355us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:13:54.548454 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0928 02:13:54.551886 (+ 3432us) server_negotiation.cc:299] Negotiation successful
0928 02:13:54.552155 (+ 269us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8047,"thread_start_us":12769,"threads_started":1}
I20260928 02:13:54.554833 19664 heartbeater.cc:347] Connected to a master server at 127.6.235.126:41799
I20260928 02:13:54.555188 19664 heartbeater.cc:464] Registering TS with master...
I20260928 02:13:54.556088 19664 heartbeater.cc:511] Master 127.6.235.126:41799 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.6.235.66@KRBTEST.COM; defaulting to no policy
I20260928 02:13:54.558890 19451 ts_manager.cc:194] Registered new tserver with Master: 269968aeb430404a8977927db1ba1660 (127.6.235.65:36397)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.235.66@KRBTEST.COM" created.
I20260928 02:13:54.561072 19451 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.235.65@KRBTEST.COM'} at 127.6.235.65:37479
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.235.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790561432251112-7085-0/krb5kdc/kudu_127.6.235.66.keytab.
Entry for principal kudu/127.6.235.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790561432251112-7085-0/krb5kdc/kudu_127.6.235.66.keytab.
I20260928 02:13:54.592378 7085 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTlaUuc/build/asan/bin/kudu
/tmp/dist-test-taskTlaUuc/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790561432251112-7085-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790561432251112-7085-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790561432251112-7085-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790561432251112-7085-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.6.235.84:35091/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.235.66:0
--local_ip_for_outbound_sockets=127.6.235.66
--webserver_interface=127.6.235.66
--webserver_port=0
--tserver_master_addrs=127.6.235.126:41799
--keytab_file=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790561432251112-7085-0/krb5kdc/kudu_127.6.235.66.keytab
--principal=kudu/127.6.235.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.235.84:40767
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790561432251112-7085-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790561432251112-7085-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790561432251112-7085-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 28 02:13:54 dist-test-slave-86gv krb5kdc[18975](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561633, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.235.126@KRBTEST.COM for HTTP/127.6.235.84@KRBTEST.COM
W20260928 02:13:55.095965 19670 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:13:55.096393 19670 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:13:55.096608 19670 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:13:55.108211 19670 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:13:55.108438 19670 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:13:55.108718 19670 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.235.66
I20260928 02:13:55.125011 19670 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.235.84:40767
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790561432251112-7085-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790561432251112-7085-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.6.235.84:35091/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790561432251112-7085-0/krb5kdc/kudu_127.6.235.66.keytab
--principal=kudu/127.6.235.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.235.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790561432251112-7085-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.6.235.66
--webserver_port=0
--tserver_master_addrs=127.6.235.126:41799
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.235.66
--log_dir=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790561432251112-7085-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 00:51:59 UTC on ccycloud.quasar-axgkwn.root.comops.site
build id 4676464
ASAN enabled
I20260928 02:13:55.127195 19670 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:13:55.129063 19670 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:13:55.144807 19685 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:13:55.144971 19684 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:13:55.145507 19687 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:13:55.146059 19670 server_base.cc:1034] running on GCE node
I20260928 02:13:55.146792 19670 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:13:55.148016 19670 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:13:55.149257 19670 hybrid_clock.cc:648] HybridClock initialized: now 1790561635149191 us; error 56 us; skew 500 ppm
I20260928 02:13:55.149652 19670 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 02:13:55 dist-test-slave-86gv krb5kdc[18975](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561635, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.235.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:13:55.153708 19670 init.cc:377] Logged in from keytab as kudu/127.6.235.66@KRBTEST.COM (short username kudu)
I20260928 02:13:55.155610 19670 webserver.cc:460] Webserver started at http://127.6.235.66:44539/ using document root <none> and password file <none>
I20260928 02:13:55.156523 19670 fs_manager.cc:359] Metadata directory not provided
I20260928 02:13:55.156665 19670 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:13:55.157087 19670 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 28 02:13:55 dist-test-slave-86gv krb5kdc[18975](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561635, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.235.66@KRBTEST.COM for HTTP/127.6.235.84@KRBTEST.COM
I20260928 02:13:55.175923 19670 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790561432251112-7085-0/minicluster-data/ts-1/data/instance:
uuid: "290fd539aafb4fceabbcd6bf66c4b1bc"
format_stamp: "Formatted at 2026-09-28 02:13:55 on dist-test-slave-86gv"
server_key: "e37e2bf3e5d3118317f226858ac2eebf"
server_key_iv: "4ef72056f3cb1e600b62605b572b891b"
server_key_version: "kuduclusterkey@0"
I20260928 02:13:55.177101 19670 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790561432251112-7085-0/minicluster-data/ts-1/wal/instance:
uuid: "290fd539aafb4fceabbcd6bf66c4b1bc"
format_stamp: "Formatted at 2026-09-28 02:13:55 on dist-test-slave-86gv"
server_key: "e37e2bf3e5d3118317f226858ac2eebf"
server_key_iv: "4ef72056f3cb1e600b62605b572b891b"
server_key_version: "kuduclusterkey@0"
I20260928 02:13:55.184386 19670 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.006s sys 0.000s
I20260928 02:13:55.203013 19694 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:13:55.204842 19670 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.000s
I20260928 02:13:55.205065 19670 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790561432251112-7085-0/minicluster-data/ts-1/data,/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790561432251112-7085-0/minicluster-data/ts-1/wal
uuid: "290fd539aafb4fceabbcd6bf66c4b1bc"
format_stamp: "Formatted at 2026-09-28 02:13:55 on dist-test-slave-86gv"
server_key: "e37e2bf3e5d3118317f226858ac2eebf"
server_key_iv: "4ef72056f3cb1e600b62605b572b891b"
server_key_version: "kuduclusterkey@0"
I20260928 02:13:55.205305 19670 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790561432251112-7085-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790561432251112-7085-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790561432251112-7085-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:13:55.227751 19670 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:13:55.233031 19670 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:13:55.233354 19670 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:13:55.234601 19670 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:13:55.236364 19670 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:13:55.236466 19670 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:13:55.236580 19670 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:13:55.236639 19670 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:13:55.278645 19670 rpc_server.cc:307] RPC server started. Bound to: 127.6.235.66:44645
I20260928 02:13:55.278669 19806 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.235.66:44645 every 8 connection(s)
I20260928 02:13:55.280288 19670 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790561432251112-7085-0/minicluster-data/ts-1/data/info.pb
I20260928 02:13:55.290402 7085 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTlaUuc/build/asan/bin/kudu as pid 19670
I20260928 02:13:55.290599 7085 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790561432251112-7085-0/minicluster-data/ts-1/wal/instance
I20260928 02:13:55.291077 7085 external_mini_cluster.cc:1428] Setting key c95401d9cff93ba93dd80cafa0e8c495
Sep 28 02:13:55 dist-test-slave-86gv krb5kdc[18975](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561635, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.235.66@KRBTEST.COM for kudu/127.6.235.126@KRBTEST.COM
I20260928 02:13:55.307834 19809 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:13:55.283081 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.235.66:37125 (local address 127.6.235.126:41799)
0928 02:13:55.283611 (+ 530us) server_negotiation.cc:206] Beginning negotiation
0928 02:13:55.283617 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 02:13:55.284815 (+ 1198us) server_negotiation.cc:407] Connection header received
0928 02:13:55.287359 (+ 2544us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:13:55.287364 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:13:55.287446 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:13:55.287544 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:13:55.289455 (+ 1911us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:13:55.290405 (+ 950us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:13:55.292800 (+ 2395us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:13:55.293285 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:13:55.293423 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:13:55.301784 (+ 8361us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:13:55.301790 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:13:55.301793 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:13:55.301827 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:13:55.304515 (+ 2688us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:13:55.305414 (+ 899us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:13:55.305418 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:13:55.305420 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:13:55.305546 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:13:55.306029 (+ 483us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:13:55.306036 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:13:55.306044 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:13:55.306334 (+ 290us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:13:55.306497 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0928 02:13:55.307219 (+ 722us) server_negotiation.cc:299] Negotiation successful
0928 02:13:55.307454 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":406,"thread_start_us":272,"threads_started":1}
I20260928 02:13:55.309072 19810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:13:55.283806 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.235.126:41799 (local address 127.6.235.66:37125)
0928 02:13:55.284612 (+ 806us) negotiation.cc:107] Waiting for socket to connect
0928 02:13:55.284666 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0928 02:13:55.287028 (+ 2362us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:13:55.287824 (+ 796us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:13:55.287835 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:13:55.288417 (+ 582us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:13:55.289270 (+ 853us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:13:55.289294 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:13:55.290576 (+ 1282us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:13:55.290581 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:13:55.292577 (+ 1996us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:13:55.292591 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:13:55.293509 (+ 918us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:13:55.293518 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:13:55.293764 (+ 246us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:13:55.299284 (+ 5520us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:13:55.299310 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:13:55.301504 (+ 2194us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:13:55.304687 (+ 3183us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:13:55.304696 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:13:55.304715 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:13:55.305271 (+ 556us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:13:55.305667 (+ 396us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:13:55.305673 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:13:55.305676 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:13:55.305903 (+ 227us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:13:55.306472 (+ 569us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:13:55.306481 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:13:55.306835 (+ 354us) client_negotiation.cc:769] Sending connection context
0928 02:13:55.307124 (+ 289us) client_negotiation.cc:240] Negotiation successful
0928 02:13:55.307682 (+ 558us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":505,"thread_start_us":305,"threads_started":1}
I20260928 02:13:55.310670 19807 heartbeater.cc:347] Connected to a master server at 127.6.235.126:41799
I20260928 02:13:55.311049 19807 heartbeater.cc:464] Registering TS with master...
I20260928 02:13:55.311976 19807 heartbeater.cc:511] Master 127.6.235.126:41799 requested a full tablet report, sending...
I20260928 02:13:55.313474 19451 ts_manager.cc:194] Registered new tserver with Master: 290fd539aafb4fceabbcd6bf66c4b1bc (127.6.235.66:44645)
I20260928 02:13:55.314422 19451 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.235.66@KRBTEST.COM'} at 127.6.235.66:37125
WARNING: no policy specified for kudu/127.6.235.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.235.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.235.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790561432251112-7085-0/krb5kdc/kudu_127.6.235.67.keytab.
Entry for principal kudu/127.6.235.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790561432251112-7085-0/krb5kdc/kudu_127.6.235.67.keytab.
I20260928 02:13:55.357239 7085 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTlaUuc/build/asan/bin/kudu
/tmp/dist-test-taskTlaUuc/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790561432251112-7085-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790561432251112-7085-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790561432251112-7085-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790561432251112-7085-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.6.235.84:35091/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.235.67:0
--local_ip_for_outbound_sockets=127.6.235.67
--webserver_interface=127.6.235.67
--webserver_port=0
--tserver_master_addrs=127.6.235.126:41799
--keytab_file=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790561432251112-7085-0/krb5kdc/kudu_127.6.235.67.keytab
--principal=kudu/127.6.235.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.235.84:40767
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790561432251112-7085-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790561432251112-7085-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790561432251112-7085-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 02:13:55.565495 19664 heartbeater.cc:503] Master 127.6.235.126:41799 was elected leader, sending a full tablet report...
W20260928 02:13:55.714430 19813 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:13:55.714738 19813 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:13:55.714839 19813 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:13:55.723064 19813 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:13:55.723203 19813 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:13:55.723332 19813 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.235.67
I20260928 02:13:55.735204 19813 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.235.84:40767
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790561432251112-7085-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790561432251112-7085-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.6.235.84:35091/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790561432251112-7085-0/krb5kdc/kudu_127.6.235.67.keytab
--principal=kudu/127.6.235.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.235.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790561432251112-7085-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.6.235.67
--webserver_port=0
--tserver_master_addrs=127.6.235.126:41799
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.235.67
--log_dir=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790561432251112-7085-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 00:51:59 UTC on ccycloud.quasar-axgkwn.root.comops.site
build id 4676464
ASAN enabled
I20260928 02:13:55.736656 19813 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:13:55.738209 19813 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:13:55.752261 19819 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:13:55.752236 19818 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:13:55.752345 19821 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:13:55.753511 19813 server_base.cc:1034] running on GCE node
I20260928 02:13:55.754271 19813 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:13:55.755318 19813 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:13:55.756588 19813 hybrid_clock.cc:648] HybridClock initialized: now 1790561635756515 us; error 84 us; skew 500 ppm
I20260928 02:13:55.757076 19813 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 02:13:55 dist-test-slave-86gv krb5kdc[18975](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561635, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.235.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:13:55.760667 19813 init.cc:377] Logged in from keytab as kudu/127.6.235.67@KRBTEST.COM (short username kudu)
I20260928 02:13:55.762495 19813 webserver.cc:460] Webserver started at http://127.6.235.67:36033/ using document root <none> and password file <none>
I20260928 02:13:55.763262 19813 fs_manager.cc:359] Metadata directory not provided
I20260928 02:13:55.763427 19813 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:13:55.763763 19813 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 28 02:13:55 dist-test-slave-86gv krb5kdc[18975](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561635, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.235.67@KRBTEST.COM for HTTP/127.6.235.84@KRBTEST.COM
I20260928 02:13:55.780799 19813 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790561432251112-7085-0/minicluster-data/ts-2/data/instance:
uuid: "ea5564b49bbf435bbb8a8008bf6b2643"
format_stamp: "Formatted at 2026-09-28 02:13:55 on dist-test-slave-86gv"
server_key: "1a83de9c040658ccc66e4181d8cd847a"
server_key_iv: "de93ba23b8d1a36fed968ca0bb21ca15"
server_key_version: "kuduclusterkey@0"
I20260928 02:13:55.781937 19813 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790561432251112-7085-0/minicluster-data/ts-2/wal/instance:
uuid: "ea5564b49bbf435bbb8a8008bf6b2643"
format_stamp: "Formatted at 2026-09-28 02:13:55 on dist-test-slave-86gv"
server_key: "1a83de9c040658ccc66e4181d8cd847a"
server_key_iv: "de93ba23b8d1a36fed968ca0bb21ca15"
server_key_version: "kuduclusterkey@0"
I20260928 02:13:55.789132 19813 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.007s sys 0.001s
I20260928 02:13:55.805188 19828 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:13:55.806820 19813 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260928 02:13:55.807010 19813 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790561432251112-7085-0/minicluster-data/ts-2/data,/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790561432251112-7085-0/minicluster-data/ts-2/wal
uuid: "ea5564b49bbf435bbb8a8008bf6b2643"
format_stamp: "Formatted at 2026-09-28 02:13:55 on dist-test-slave-86gv"
server_key: "1a83de9c040658ccc66e4181d8cd847a"
server_key_iv: "de93ba23b8d1a36fed968ca0bb21ca15"
server_key_version: "kuduclusterkey@0"
I20260928 02:13:55.807257 19813 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790561432251112-7085-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790561432251112-7085-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790561432251112-7085-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:13:55.840488 19813 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:13:55.844744 19813 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:13:55.845183 19813 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:13:55.846392 19813 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:13:55.847896 19813 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:13:55.847973 19813 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:13:55.848117 19813 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:13:55.848169 19813 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:13:55.881544 19813 rpc_server.cc:307] RPC server started. Bound to: 127.6.235.67:43127
I20260928 02:13:55.881589 19940 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.235.67:43127 every 8 connection(s)
I20260928 02:13:55.883196 19813 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790561432251112-7085-0/minicluster-data/ts-2/data/info.pb
I20260928 02:13:55.886858 7085 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTlaUuc/build/asan/bin/kudu as pid 19813
I20260928 02:13:55.887032 7085 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790561432251112-7085-0/minicluster-data/ts-2/wal/instance
I20260928 02:13:55.887651 7085 external_mini_cluster.cc:1428] Setting key 30a9f4b62e2c72e6ec446babf2e7ae50
Sep 28 02:13:55 dist-test-slave-86gv krb5kdc[18975](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561635, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.235.67@KRBTEST.COM for kudu/127.6.235.126@KRBTEST.COM
I20260928 02:13:55.903318 19943 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:13:55.885790 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.235.67:57043 (local address 127.6.235.126:41799)
0928 02:13:55.886310 (+ 520us) server_negotiation.cc:206] Beginning negotiation
0928 02:13:55.886314 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:13:55.887314 (+ 1000us) server_negotiation.cc:407] Connection header received
0928 02:13:55.889373 (+ 2059us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:13:55.889378 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:13:55.889468 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:13:55.889588 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:13:55.891371 (+ 1783us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:13:55.892122 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:13:55.893704 (+ 1582us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:13:55.894137 (+ 433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:13:55.894227 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:13:55.897446 (+ 3219us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:13:55.897451 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:13:55.897454 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:13:55.897484 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:13:55.899964 (+ 2480us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:13:55.900764 (+ 800us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:13:55.900769 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:13:55.900774 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:13:55.900959 (+ 185us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:13:55.901471 (+ 512us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:13:55.901474 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:13:55.901477 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:13:55.901773 (+ 296us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:13:55.902008 (+ 235us) server_negotiation.cc:1035] Waiting for connection context
0928 02:13:55.902764 (+ 756us) server_negotiation.cc:299] Negotiation successful
0928 02:13:55.902953 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":423,"thread_start_us":248,"threads_started":1}
I20260928 02:13:55.904225 19944 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:13:55.886266 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.235.126:41799 (local address 127.6.235.67:57043)
0928 02:13:55.887141 (+ 875us) negotiation.cc:107] Waiting for socket to connect
0928 02:13:55.887195 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0928 02:13:55.889116 (+ 1921us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:13:55.889899 (+ 783us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:13:55.889911 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:13:55.890491 (+ 580us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:13:55.891216 (+ 725us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:13:55.891235 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:13:55.892253 (+ 1018us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:13:55.892257 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:13:55.893545 (+ 1288us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:13:55.893595 (+ 50us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:13:55.894310 (+ 715us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:13:55.894314 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:13:55.894509 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:13:55.895150 (+ 641us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:13:55.895171 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:13:55.897221 (+ 2050us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:13:55.900188 (+ 2967us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:13:55.900197 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:13:55.900220 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:13:55.900637 (+ 417us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:13:55.901092 (+ 455us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:13:55.901098 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:13:55.901102 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:13:55.901335 (+ 233us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:13:55.901885 (+ 550us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:13:55.901895 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:13:55.902297 (+ 402us) client_negotiation.cc:769] Sending connection context
0928 02:13:55.902608 (+ 311us) client_negotiation.cc:240] Negotiation successful
0928 02:13:55.903098 (+ 490us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":572,"thread_start_us":335,"threads_started":1}
I20260928 02:13:55.905592 19941 heartbeater.cc:347] Connected to a master server at 127.6.235.126:41799
I20260928 02:13:55.905895 19941 heartbeater.cc:464] Registering TS with master...
I20260928 02:13:55.906639 19941 heartbeater.cc:511] Master 127.6.235.126:41799 requested a full tablet report, sending...
I20260928 02:13:55.908078 19451 ts_manager.cc:194] Registered new tserver with Master: ea5564b49bbf435bbb8a8008bf6b2643 (127.6.235.67:43127)
I20260928 02:13:55.908874 19451 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.235.67@KRBTEST.COM'} at 127.6.235.67:57043
I20260928 02:13:55.915551 7085 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 02:13:55.917312 7085 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskTlaUuc/build/asan/bin/kudu pid 19531
Sep 28 02:13:55 dist-test-slave-86gv krb5kdc[18975](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561632, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.235.65@KRBTEST.COM
I20260928 02:13:55.934719 19946 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:13:55.918198 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40658 (local address 127.6.235.65:36397)
0928 02:13:55.918786 (+ 588us) server_negotiation.cc:206] Beginning negotiation
0928 02:13:55.918799 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0928 02:13:55.918833 (+ 34us) server_negotiation.cc:407] Connection header received
0928 02:13:55.919067 (+ 234us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:13:55.919078 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:13:55.919301 (+ 223us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:13:55.919479 (+ 178us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:13:55.920980 (+ 1501us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:13:55.921990 (+ 1010us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:13:55.923539 (+ 1549us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:13:55.924114 (+ 575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:13:55.924258 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:13:55.928619 (+ 4361us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:13:55.928634 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:13:55.928659 (+ 25us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:13:55.928697 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:13:55.931531 (+ 2834us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:13:55.932326 (+ 795us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:13:55.932341 (+ 15us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:13:55.932344 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:13:55.932486 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:13:55.932887 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:13:55.932918 (+ 31us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:13:55.932921 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:13:55.933285 (+ 364us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:13:55.933615 (+ 330us) server_negotiation.cc:1035] Waiting for connection context
0928 02:13:55.934025 (+ 410us) server_negotiation.cc:299] Negotiation successful
0928 02:13:55.934354 (+ 329us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":430,"thread_start_us":272,"threads_started":1}
I20260928 02:13:55.936090 19659 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 02:13:56.022089 7085 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTlaUuc/build/asan/bin/kudu with pid 19531
I20260928 02:13:56.039482 7085 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskTlaUuc/build/asan/bin/kudu pid 19670
Sep 28 02:13:56 dist-test-slave-86gv krb5kdc[18975](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561632, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.235.66@KRBTEST.COM
I20260928 02:13:56.056701 19948 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:13:56.040491 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55054 (local address 127.6.235.66:44645)
0928 02:13:56.041080 (+ 589us) server_negotiation.cc:206] Beginning negotiation
0928 02:13:56.041088 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 02:13:56.041115 (+ 27us) server_negotiation.cc:407] Connection header received
0928 02:13:56.041175 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:13:56.041181 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:13:56.041418 (+ 237us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:13:56.041562 (+ 144us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:13:56.042768 (+ 1206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:13:56.043757 (+ 989us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:13:56.045131 (+ 1374us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:13:56.045750 (+ 619us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:13:56.045860 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:13:56.050789 (+ 4929us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:13:56.050804 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:13:56.050819 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:13:56.050858 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:13:56.053983 (+ 3125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:13:56.054743 (+ 760us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:13:56.054750 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:13:56.054752 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:13:56.054882 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:13:56.055217 (+ 335us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:13:56.055221 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:13:56.055223 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:13:56.055584 (+ 361us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:13:56.055806 (+ 222us) server_negotiation.cc:1035] Waiting for connection context
0928 02:13:56.056060 (+ 254us) server_negotiation.cc:299] Negotiation successful
0928 02:13:56.056267 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":438,"thread_start_us":263,"threads_started":1}
I20260928 02:13:56.058120 19802 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 02:13:56.143148 7085 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTlaUuc/build/asan/bin/kudu with pid 19670
I20260928 02:13:56.163897 7085 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskTlaUuc/build/asan/bin/kudu pid 19813
Sep 28 02:13:56 dist-test-slave-86gv krb5kdc[18975](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561632, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.235.67@KRBTEST.COM
I20260928 02:13:56.181197 19950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:13:56.165088 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49788 (local address 127.6.235.67:43127)
0928 02:13:56.165766 (+ 678us) server_negotiation.cc:206] Beginning negotiation
0928 02:13:56.165773 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 02:13:56.165803 (+ 30us) server_negotiation.cc:407] Connection header received
0928 02:13:56.165861 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:13:56.165868 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:13:56.166083 (+ 215us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:13:56.166226 (+ 143us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:13:56.167445 (+ 1219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:13:56.168426 (+ 981us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:13:56.169805 (+ 1379us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:13:56.170355 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:13:56.170455 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:13:56.174736 (+ 4281us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:13:56.174748 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:13:56.174763 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:13:56.174801 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:13:56.177820 (+ 3019us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:13:56.178747 (+ 927us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:13:56.178755 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:13:56.178757 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:13:56.178900 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:13:56.179517 (+ 617us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:13:56.179520 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:13:56.179522 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:13:56.179901 (+ 379us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:13:56.180168 (+ 267us) server_negotiation.cc:1035] Waiting for connection context
0928 02:13:56.180489 (+ 321us) server_negotiation.cc:299] Negotiation successful
0928 02:13:56.180696 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":529,"thread_start_us":352,"threads_started":1}
I20260928 02:13:56.182699 19936 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 02:13:56.266947 7085 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTlaUuc/build/asan/bin/kudu with pid 19813
I20260928 02:13:56.284418 7085 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskTlaUuc/build/asan/bin/kudu pid 19419
I20260928 02:13:56.285893 19481 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 02:13:56.393280 7085 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTlaUuc/build/asan/bin/kudu with pid 19419
I20260928 02:13:56.417398 7085 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTlaUuc/build/asan/bin/kudu
/tmp/dist-test-taskTlaUuc/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790561432251112-7085-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790561432251112-7085-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790561432251112-7085-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790561432251112-7085-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.235.126:41799
--webserver_interface=127.6.235.126
--webserver_port=41575
--keytab_file=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790561432251112-7085-0/krb5kdc/kudu_127.6.235.126.keytab
--principal=kudu/127.6.235.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.235.84:40767
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.235.126:41799
--ranger_config_path=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790561432251112-7085-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.6.235.84:35091/kms
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.6.235.84:35091/kms with env {KRB5CCNAME=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790561432251112-7085-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790561432251112-7085-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790561432251112-7085-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:13:56.760959 19953 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:13:56.761307 19953 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:13:56.761469 19953 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:13:56.768829 19953 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 02:13:56.769029 19953 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:13:56.769148 19953 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:13:56.769238 19953 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 02:13:56.769305 19953 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 02:13:56.779824 19953 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.235.84:40767
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790561432251112-7085-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790561432251112-7085-0/minicluster-data/master-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.6.235.84:35091/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.6.235.126:41799
--ranger_config_path=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790561432251112-7085-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790561432251112-7085-0/krb5kdc/kudu_127.6.235.126.keytab
--principal=kudu/127.6.235.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.235.126:41799
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790561432251112-7085-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.235.126
--webserver_port=41575
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790561432251112-7085-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 00:51:59 UTC on ccycloud.quasar-axgkwn.root.comops.site
build id 4676464
ASAN enabled
I20260928 02:13:56.781224 19953 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:13:56.782526 19953 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:13:56.794664 19958 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:13:56.794749 19959 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:13:56.794687 19961 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:13:56.795378 19953 server_base.cc:1034] running on GCE node
I20260928 02:13:56.796109 19953 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:13:56.797660 19953 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:13:56.798869 19953 hybrid_clock.cc:648] HybridClock initialized: now 1790561636798834 us; error 37 us; skew 500 ppm
I20260928 02:13:56.799197 19953 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 02:13:56 dist-test-slave-86gv krb5kdc[18975](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561636, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.235.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:13:56.802312 19953 init.cc:377] Logged in from keytab as kudu/127.6.235.126@KRBTEST.COM (short username kudu)
I20260928 02:13:56.803890 19953 webserver.cc:460] Webserver started at http://127.6.235.126:41575/ using document root <none> and password file <none>
I20260928 02:13:56.804579 19953 fs_manager.cc:359] Metadata directory not provided
I20260928 02:13:56.804733 19953 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 28 02:13:56 dist-test-slave-86gv krb5kdc[18975](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561636, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.235.126@KRBTEST.COM for HTTP/127.6.235.84@KRBTEST.COM
I20260928 02:13:56.869151 19953 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20260928 02:13:56.872290 19969 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:13:56.873863 19953 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 02:13:56.874116 19953 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790561432251112-7085-0/minicluster-data/master-0/data,/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790561432251112-7085-0/minicluster-data/master-0/wal
uuid: "5948defd0e5e4ca39ca7d4f2883db7eb"
format_stamp: "Formatted at 2026-09-28 02:13:52 on dist-test-slave-86gv"
server_key: "9db666a31f29b0944d608b8a612f51e6"
server_key_iv: "f0b95912fc1bb4991cd3fb265bba55df"
server_key_version: "kuduclusterkey@0"
I20260928 02:13:56.875222 19953 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790561432251112-7085-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790561432251112-7085-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790561432251112-7085-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:13:56.898299 19953 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:13:56.902313 19953 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:13:56.902626 19953 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:13:56.922430 19953 rpc_server.cc:307] RPC server started. Bound to: 127.6.235.126:41799
I20260928 02:13:56.922451 20020 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.235.126:41799 every 8 connection(s)
I20260928 02:13:56.924098 19953 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790561432251112-7085-0/minicluster-data/master-0/data/info.pb
I20260928 02:13:56.931334 20021 sys_catalog.cc:263] Verifying existing consensus state
I20260928 02:13:56.932574 7085 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTlaUuc/build/asan/bin/kudu as pid 19953
I20260928 02:13:56.933558 7085 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTlaUuc/build/asan/bin/kudu
/tmp/dist-test-taskTlaUuc/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790561432251112-7085-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790561432251112-7085-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790561432251112-7085-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790561432251112-7085-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.6.235.84:35091/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.235.65:36397
--local_ip_for_outbound_sockets=127.6.235.65
--tserver_master_addrs=127.6.235.126:41799
--webserver_port=40937
--webserver_interface=127.6.235.65
--keytab_file=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790561432251112-7085-0/krb5kdc/kudu_127.6.235.65.keytab
--principal=kudu/127.6.235.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.235.84:40767
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.6.235.84:35091/kms with env {KRB5CCNAME=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790561432251112-7085-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790561432251112-7085-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790561432251112-7085-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 02:13:56.934911 20021 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5948defd0e5e4ca39ca7d4f2883db7eb: Bootstrap starting.
I20260928 02:13:56.954246 20021 log.cc:824] T 00000000000000000000000000000000 P 5948defd0e5e4ca39ca7d4f2883db7eb: Log is configured to *not* fsync() on all Append() calls
I20260928 02:13:56.963500 20021 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5948defd0e5e4ca39ca7d4f2883db7eb: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 02:13:56.964048 20021 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5948defd0e5e4ca39ca7d4f2883db7eb: Bootstrap complete.
I20260928 02:13:56.973105 20021 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5948defd0e5e4ca39ca7d4f2883db7eb [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5948defd0e5e4ca39ca7d4f2883db7eb" member_type: VOTER last_known_addr { host: "127.6.235.126" port: 41799 } }
I20260928 02:13:56.974022 20021 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5948defd0e5e4ca39ca7d4f2883db7eb [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5948defd0e5e4ca39ca7d4f2883db7eb, State: Initialized, Role: FOLLOWER
I20260928 02:13:56.974637 20021 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5948defd0e5e4ca39ca7d4f2883db7eb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5948defd0e5e4ca39ca7d4f2883db7eb" member_type: VOTER last_known_addr { host: "127.6.235.126" port: 41799 } }
I20260928 02:13:56.974881 20021 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5948defd0e5e4ca39ca7d4f2883db7eb [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 02:13:56.975015 20021 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5948defd0e5e4ca39ca7d4f2883db7eb [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 02:13:56.975201 20021 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5948defd0e5e4ca39ca7d4f2883db7eb [term 1 FOLLOWER]: Advancing to term 2
I20260928 02:13:56.979279 20021 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5948defd0e5e4ca39ca7d4f2883db7eb [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5948defd0e5e4ca39ca7d4f2883db7eb" member_type: VOTER last_known_addr { host: "127.6.235.126" port: 41799 } }
I20260928 02:13:56.979849 20021 leader_election.cc:302] T 00000000000000000000000000000000 P 5948defd0e5e4ca39ca7d4f2883db7eb [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5948defd0e5e4ca39ca7d4f2883db7eb; no voters:
I20260928 02:13:56.980532 20021 leader_election.cc:288] T 00000000000000000000000000000000 P 5948defd0e5e4ca39ca7d4f2883db7eb [CANDIDATE]: Term 2 election: Requested vote from peers
I20260928 02:13:56.980675 20025 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5948defd0e5e4ca39ca7d4f2883db7eb [term 2 FOLLOWER]: Leader election won for term 2
I20260928 02:13:56.983084 20021 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5948defd0e5e4ca39ca7d4f2883db7eb [sys.catalog]: configured and running, proceeding with master startup.
I20260928 02:13:56.985370 20025 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5948defd0e5e4ca39ca7d4f2883db7eb [term 2 LEADER]: Becoming Leader. State: Replica: 5948defd0e5e4ca39ca7d4f2883db7eb, State: Running, Role: LEADER
I20260928 02:13:56.986135 20025 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5948defd0e5e4ca39ca7d4f2883db7eb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5948defd0e5e4ca39ca7d4f2883db7eb" member_type: VOTER last_known_addr { host: "127.6.235.126" port: 41799 } }
I20260928 02:13:56.990599 20026 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5948defd0e5e4ca39ca7d4f2883db7eb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "5948defd0e5e4ca39ca7d4f2883db7eb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5948defd0e5e4ca39ca7d4f2883db7eb" member_type: VOTER last_known_addr { host: "127.6.235.126" port: 41799 } } }
I20260928 02:13:56.990676 20027 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5948defd0e5e4ca39ca7d4f2883db7eb [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5948defd0e5e4ca39ca7d4f2883db7eb. Latest consensus state: current_term: 2 leader_uuid: "5948defd0e5e4ca39ca7d4f2883db7eb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5948defd0e5e4ca39ca7d4f2883db7eb" member_type: VOTER last_known_addr { host: "127.6.235.126" port: 41799 } } }
I20260928 02:13:56.991005 20026 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5948defd0e5e4ca39ca7d4f2883db7eb [sys.catalog]: This master's current role is: LEADER
I20260928 02:13:56.991093 20027 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5948defd0e5e4ca39ca7d4f2883db7eb [sys.catalog]: This master's current role is: LEADER
I20260928 02:13:57.005196 20036 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 02:13:57.011042 20036 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 02:13:57.011870 20021 ranger_client.cc:318] Using overwritten properties file: /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790561432251112-7085-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260928 02:13:57.012584 20036 catalog_manager.cc:1259] Loaded cluster ID: 58fbf9e1b9bb455ca74c27c45b660094
I20260928 02:13:57.012692 20036 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 02:13:57.016698 20036 catalog_manager.cc:1504] Loading token signing keys...
I20260928 02:13:57.018160 20036 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 5948defd0e5e4ca39ca7d4f2883db7eb: Loaded TSK: 0
I20260928 02:13:57.019744 20036 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260928 02:13:57.379835 20023 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:13:57.380362 20023 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:13:57.380575 20023 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:13:57.391049 20023 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:13:57.391271 20023 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:13:57.391500 20023 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.235.65
I20260928 02:13:57.404850 20023 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.235.84:40767
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790561432251112-7085-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790561432251112-7085-0/minicluster-data/ts-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.6.235.84:35091/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790561432251112-7085-0/krb5kdc/kudu_127.6.235.65.keytab
--principal=kudu/127.6.235.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.235.65:36397
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790561432251112-7085-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.6.235.65
--webserver_port=40937
--tserver_master_addrs=127.6.235.126:41799
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.235.65
--log_dir=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790561432251112-7085-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 00:51:59 UTC on ccycloud.quasar-axgkwn.root.comops.site
build id 4676464
ASAN enabled
I20260928 02:13:57.406868 20023 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:13:57.408576 20023 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:13:57.436092 20058 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:13:57.438601 20059 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:13:57.445140 20023 server_base.cc:1034] running on GCE node
W20260928 02:13:57.446336 20061 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:13:57.447247 20023 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:13:57.457309 20023 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:13:57.458601 20023 hybrid_clock.cc:648] HybridClock initialized: now 1790561637458525 us; error 79 us; skew 500 ppm
I20260928 02:13:57.459017 20023 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 02:13:57 dist-test-slave-86gv krb5kdc[18975](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561637, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.235.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:13:57.463086 20023 init.cc:377] Logged in from keytab as kudu/127.6.235.65@KRBTEST.COM (short username kudu)
I20260928 02:13:57.470546 20023 webserver.cc:460] Webserver started at http://127.6.235.65:40937/ using document root <none> and password file <none>
I20260928 02:13:57.471731 20023 fs_manager.cc:359] Metadata directory not provided
I20260928 02:13:57.471964 20023 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 28 02:13:57 dist-test-slave-86gv krb5kdc[18975](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561637, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.235.65@KRBTEST.COM for HTTP/127.6.235.84@KRBTEST.COM
I20260928 02:13:57.503654 20023 fs_manager.cc:709] Time spent opening directory manager: real 0.011s user 0.007s sys 0.003s
I20260928 02:13:57.514441 20069 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:13:57.516629 20023 fs_manager.cc:725] Time spent opening block manager: real 0.011s user 0.005s sys 0.004s
I20260928 02:13:57.516928 20023 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790561432251112-7085-0/minicluster-data/ts-0/data,/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790561432251112-7085-0/minicluster-data/ts-0/wal
uuid: "269968aeb430404a8977927db1ba1660"
format_stamp: "Formatted at 2026-09-28 02:13:54 on dist-test-slave-86gv"
server_key: "1920d280ef95171b6b956cd0b5f379d9"
server_key_iv: "efa66e220e9b8791b81d9266138266b8"
server_key_version: "kuduclusterkey@0"
I20260928 02:13:57.518508 20023 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790561432251112-7085-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790561432251112-7085-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790561432251112-7085-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:13:57.572775 20023 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:13:57.578660 20023 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:13:57.579051 20023 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:13:57.586863 20023 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:13:57.594094 20023 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:13:57.594307 20023 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.001s sys 0.000s
I20260928 02:13:57.594514 20023 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:13:57.594655 20023 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:13:57.793162 20023 rpc_server.cc:307] RPC server started. Bound to: 127.6.235.65:36397
I20260928 02:13:57.793206 20182 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.235.65:36397 every 8 connection(s)
I20260928 02:13:57.794754 20023 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790561432251112-7085-0/minicluster-data/ts-0/data/info.pb
I20260928 02:13:57.805303 7085 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTlaUuc/build/asan/bin/kudu as pid 20023
I20260928 02:13:57.806378 7085 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTlaUuc/build/asan/bin/kudu
/tmp/dist-test-taskTlaUuc/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790561432251112-7085-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790561432251112-7085-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790561432251112-7085-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790561432251112-7085-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.6.235.84:35091/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.235.66:44645
--local_ip_for_outbound_sockets=127.6.235.66
--tserver_master_addrs=127.6.235.126:41799
--webserver_port=44539
--webserver_interface=127.6.235.66
--keytab_file=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790561432251112-7085-0/krb5kdc/kudu_127.6.235.66.keytab
--principal=kudu/127.6.235.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.235.84:40767
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.6.235.84:35091/kms with env {KRB5CCNAME=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790561432251112-7085-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790561432251112-7085-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790561432251112-7085-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 28 02:13:57 dist-test-slave-86gv krb5kdc[18975](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561637, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.235.65@KRBTEST.COM for kudu/127.6.235.126@KRBTEST.COM
I20260928 02:13:57.835985 20186 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:13:57.801273 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.235.126:41799 (local address 127.6.235.65:38913)
0928 02:13:57.805724 (+ 4451us) negotiation.cc:107] Waiting for socket to connect
0928 02:13:57.805775 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0928 02:13:57.808520 (+ 2745us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:13:57.812008 (+ 3488us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:13:57.812021 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:13:57.812716 (+ 695us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:13:57.813681 (+ 965us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:13:57.813711 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:13:57.816341 (+ 2630us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:13:57.816348 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:13:57.818426 (+ 2078us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:13:57.818456 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:13:57.820455 (+ 1999us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:13:57.820465 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:13:57.820781 (+ 316us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:13:57.821722 (+ 941us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:13:57.821758 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:13:57.824631 (+ 2873us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:13:57.830176 (+ 5545us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:13:57.830187 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:13:57.830205 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:13:57.830881 (+ 676us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:13:57.831529 (+ 648us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:13:57.831534 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:13:57.831538 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:13:57.831842 (+ 304us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:13:57.832981 (+ 1139us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:13:57.832990 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:13:57.833446 (+ 456us) client_negotiation.cc:769] Sending connection context
0928 02:13:57.833789 (+ 343us) client_negotiation.cc:240] Negotiation successful
0928 02:13:57.834411 (+ 622us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4137,"thread_start_us":3933,"threads_started":1}
I20260928 02:13:57.839625 20185 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:13:57.801076 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.235.65:38913 (local address 127.6.235.126:41799)
0928 02:13:57.809543 (+ 8467us) server_negotiation.cc:206] Beginning negotiation
0928 02:13:57.809566 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0928 02:13:57.809621 (+ 55us) server_negotiation.cc:407] Connection header received
0928 02:13:57.810090 (+ 469us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:13:57.810113 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:13:57.810696 (+ 583us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:13:57.811177 (+ 481us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:13:57.813968 (+ 2791us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:13:57.816116 (+ 2148us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:13:57.818716 (+ 2600us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:13:57.819773 (+ 1057us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:13:57.819979 (+ 206us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:13:57.825566 (+ 5587us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:13:57.825579 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:13:57.825603 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:13:57.825661 (+ 58us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:13:57.829956 (+ 4295us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:13:57.831127 (+ 1171us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:13:57.831138 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:13:57.831146 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:13:57.831373 (+ 227us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:13:57.831995 (+ 622us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:13:57.832001 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:13:57.832004 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:13:57.832756 (+ 752us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:13:57.837277 (+ 4521us) server_negotiation.cc:1035] Waiting for connection context
0928 02:13:57.837639 (+ 362us) server_negotiation.cc:299] Negotiation successful
0928 02:13:57.838129 (+ 490us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8177,"thread_start_us":7896,"threads_started":1}
I20260928 02:13:57.843961 20183 heartbeater.cc:347] Connected to a master server at 127.6.235.126:41799
I20260928 02:13:57.844326 20183 heartbeater.cc:464] Registering TS with master...
I20260928 02:13:57.845328 20183 heartbeater.cc:511] Master 127.6.235.126:41799 requested a full tablet report, sending...
W20260928 02:13:57.848978 20183 heartbeater.cc:644] Failed to heartbeat to 127.6.235.126:41799 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260928 02:13:57.849241 20183 heartbeater.cc:464] Registering TS with master...
I20260928 02:13:57.849571 20183 heartbeater.cc:511] Master 127.6.235.126:41799 requested a full tablet report, sending...
I20260928 02:13:57.851253 20183 heartbeater.cc:464] Registering TS with master...
I20260928 02:13:57.851719 20183 heartbeater.cc:511] Master 127.6.235.126:41799 requested a full tablet report, sending...
W20260928 02:13:57.853127 20183 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260928 02:13:58.330543 20187 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:13:58.330965 20187 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:13:58.331178 20187 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:13:58.343911 20187 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:13:58.344102 20187 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:13:58.344285 20187 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.235.66
I20260928 02:13:58.359508 20187 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.235.84:40767
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790561432251112-7085-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790561432251112-7085-0/minicluster-data/ts-1/wal
--ranger_kms_url=invalid.host:1234/kms,127.6.235.84:35091/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790561432251112-7085-0/krb5kdc/kudu_127.6.235.66.keytab
--principal=kudu/127.6.235.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.235.66:44645
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790561432251112-7085-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.6.235.66
--webserver_port=44539
--tserver_master_addrs=127.6.235.126:41799
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.235.66
--log_dir=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790561432251112-7085-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 00:51:59 UTC on ccycloud.quasar-axgkwn.root.comops.site
build id 4676464
ASAN enabled
I20260928 02:13:58.361268 20187 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:13:58.363059 20187 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:13:58.386876 20202 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:13:58.393290 20201 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:13:58.395741 20204 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:13:58.397408 20187 server_base.cc:1034] running on GCE node
I20260928 02:13:58.398213 20187 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:13:58.414125 20187 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:13:58.419884 20187 hybrid_clock.cc:648] HybridClock initialized: now 1790561638419758 us; error 125 us; skew 500 ppm
I20260928 02:13:58.420488 20187 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 02:13:58 dist-test-slave-86gv krb5kdc[18975](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561638, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.235.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:13:58.426136 20187 init.cc:377] Logged in from keytab as kudu/127.6.235.66@KRBTEST.COM (short username kudu)
I20260928 02:13:58.428207 20187 webserver.cc:460] Webserver started at http://127.6.235.66:44539/ using document root <none> and password file <none>
I20260928 02:13:58.429098 20187 fs_manager.cc:359] Metadata directory not provided
I20260928 02:13:58.429334 20187 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 28 02:13:58 dist-test-slave-86gv krb5kdc[18975](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561638, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.235.66@KRBTEST.COM for HTTP/127.6.235.84@KRBTEST.COM
I20260928 02:13:58.461910 20187 fs_manager.cc:709] Time spent opening directory manager: real 0.007s user 0.005s sys 0.001s
I20260928 02:13:58.468654 20214 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:13:58.470783 20187 fs_manager.cc:725] Time spent opening block manager: real 0.007s user 0.006s sys 0.001s
I20260928 02:13:58.470995 20187 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790561432251112-7085-0/minicluster-data/ts-1/data,/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790561432251112-7085-0/minicluster-data/ts-1/wal
uuid: "290fd539aafb4fceabbcd6bf66c4b1bc"
format_stamp: "Formatted at 2026-09-28 02:13:55 on dist-test-slave-86gv"
server_key: "e37e2bf3e5d3118317f226858ac2eebf"
server_key_iv: "4ef72056f3cb1e600b62605b572b891b"
server_key_version: "kuduclusterkey@0"
I20260928 02:13:58.472420 20187 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790561432251112-7085-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790561432251112-7085-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790561432251112-7085-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
Sep 28 02:13:58 dist-test-slave-86gv krb5kdc[18975](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790561638, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.235.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:13:58.506590 20187 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:13:58.512392 20187 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:13:58.512791 20187 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:13:58.514147 20187 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:13:58.519536 20187 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:13:58.519722 20187 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.001s sys 0.000s
I20260928 02:13:58.519878 20187 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:13:58.520012 20187 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:13:58.603431 20187 rpc_server.cc:307] RPC server started. Bound to: 127.6.235.66:44645
I20260928 02:13:58.603514 20327 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.235.66:44645 every 8 connection(s)
I20260928 02:13:58.605636 20187 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790561432251112-7085-0/minicluster-data/ts-1/data/info.pb
I20260928 02:13:58.613683 7085 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTlaUuc/build/asan/bin/kudu as pid 20187
I20260928 02:13:58.614501 7085 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTlaUuc/build/asan/bin/kudu
/tmp/dist-test-taskTlaUuc/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790561432251112-7085-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790561432251112-7085-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790561432251112-7085-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790561432251112-7085-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.6.235.84:35091/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.235.67:43127
--local_ip_for_outbound_sockets=127.6.235.67
--tserver_master_addrs=127.6.235.126:41799
--webserver_port=36033
--webserver_interface=127.6.235.67
--keytab_file=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790561432251112-7085-0/krb5kdc/kudu_127.6.235.67.keytab
--principal=kudu/127.6.235.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.235.84:40767
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.6.235.84:35091/kms with env {KRB5CCNAME=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790561432251112-7085-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790561432251112-7085-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790561432251112-7085-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 28 02:13:58 dist-test-slave-86gv krb5kdc[18975](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561638, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.235.66@KRBTEST.COM for kudu/127.6.235.126@KRBTEST.COM
I20260928 02:13:58.640844 20331 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:13:58.610392 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.235.66:34061 (local address 127.6.235.126:41799)
0928 02:13:58.617461 (+ 7069us) server_negotiation.cc:206] Beginning negotiation
0928 02:13:58.617469 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 02:13:58.621155 (+ 3686us) server_negotiation.cc:407] Connection header received
0928 02:13:58.621235 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:13:58.621240 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:13:58.621334 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:13:58.621441 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:13:58.623955 (+ 2514us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:13:58.625189 (+ 1234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:13:58.627707 (+ 2518us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:13:58.628373 (+ 666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:13:58.628506 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:13:58.633291 (+ 4785us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:13:58.633299 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:13:58.633304 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:13:58.633350 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:13:58.637007 (+ 3657us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:13:58.637916 (+ 909us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:13:58.637920 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:13:58.637922 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:13:58.638044 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:13:58.638776 (+ 732us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:13:58.638783 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:13:58.638787 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:13:58.639102 (+ 315us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:13:58.639289 (+ 187us) server_negotiation.cc:1035] Waiting for connection context
0928 02:13:58.640172 (+ 883us) server_negotiation.cc:299] Negotiation successful
0928 02:13:58.640454 (+ 282us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6921,"thread_start_us":6790,"threads_started":1}
I20260928 02:13:58.642203 20330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:13:58.609679 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.235.126:41799 (local address 127.6.235.66:34061)
0928 02:13:58.618069 (+ 8390us) negotiation.cc:107] Waiting for socket to connect
0928 02:13:58.618134 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0928 02:13:58.620682 (+ 2548us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:13:58.621917 (+ 1235us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:13:58.621942 (+ 25us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:13:58.622790 (+ 848us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:13:58.623708 (+ 918us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:13:58.623736 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:13:58.625382 (+ 1646us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:13:58.625388 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:13:58.627498 (+ 2110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:13:58.627524 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:13:58.628829 (+ 1305us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:13:58.628837 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:13:58.629168 (+ 331us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:13:58.630027 (+ 859us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:13:58.630057 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:13:58.632970 (+ 2913us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:13:58.637196 (+ 4226us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:13:58.637212 (+ 16us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:13:58.637227 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:13:58.637731 (+ 504us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:13:58.638170 (+ 439us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:13:58.638175 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:13:58.638179 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:13:58.638395 (+ 216us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:13:58.639247 (+ 852us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:13:58.639256 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:13:58.639707 (+ 451us) client_negotiation.cc:769] Sending connection context
0928 02:13:58.640079 (+ 372us) client_negotiation.cc:240] Negotiation successful
0928 02:13:58.640686 (+ 607us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8032,"thread_start_us":10372,"threads_started":1}
I20260928 02:13:58.644150 20328 heartbeater.cc:347] Connected to a master server at 127.6.235.126:41799
I20260928 02:13:58.644493 20328 heartbeater.cc:464] Registering TS with master...
I20260928 02:13:58.645470 20328 heartbeater.cc:511] Master 127.6.235.126:41799 requested a full tablet report, sending...
I20260928 02:13:58.647836 19986 ts_manager.cc:194] Registered new tserver with Master: 290fd539aafb4fceabbcd6bf66c4b1bc (127.6.235.66:44645)
I20260928 02:13:58.653460 19986 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.235.66@KRBTEST.COM'} at 127.6.235.66:34061
I20260928 02:13:58.853276 20183 heartbeater.cc:464] Registering TS with master...
I20260928 02:13:58.853789 20183 heartbeater.cc:511] Master 127.6.235.126:41799 requested a full tablet report, sending...
I20260928 02:13:58.855127 19986 ts_manager.cc:194] Registered new tserver with Master: 269968aeb430404a8977927db1ba1660 (127.6.235.65:36397)
I20260928 02:13:58.856320 19986 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.235.65@KRBTEST.COM'} at 127.6.235.65:38913
Sep 28 02:13:58 dist-test-slave-86gv krb5kdc[18975](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561638, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.235.126@KRBTEST.COM for HTTP/127.6.235.84@KRBTEST.COM
W20260928 02:13:59.051986 20332 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:13:59.052310 20332 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:13:59.052475 20332 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:13:59.061362 20332 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:13:59.061506 20332 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:13:59.061728 20332 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.235.67
I20260928 02:13:59.076086 20332 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.235.84:40767
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790561432251112-7085-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790561432251112-7085-0/minicluster-data/ts-2/wal
--ranger_kms_url=invalid.host:1234/kms,127.6.235.84:35091/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790561432251112-7085-0/krb5kdc/kudu_127.6.235.67.keytab
--principal=kudu/127.6.235.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.235.67:43127
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790561432251112-7085-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.6.235.67
--webserver_port=36033
--tserver_master_addrs=127.6.235.126:41799
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.235.67
--log_dir=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790561432251112-7085-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 00:51:59 UTC on ccycloud.quasar-axgkwn.root.comops.site
build id 4676464
ASAN enabled
I20260928 02:13:59.078040 20332 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:13:59.079829 20332 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:13:59.097699 20337 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:13:59.098312 20338 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:13:59.100304 20340 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:13:59.101006 20332 server_base.cc:1034] running on GCE node
I20260928 02:13:59.101672 20332 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:13:59.102762 20332 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:13:59.104164 20332 hybrid_clock.cc:648] HybridClock initialized: now 1790561639104090 us; error 50 us; skew 500 ppm
I20260928 02:13:59.104525 20332 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 02:13:59 dist-test-slave-86gv krb5kdc[18975](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561639, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.235.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:13:59.107921 20332 init.cc:377] Logged in from keytab as kudu/127.6.235.67@KRBTEST.COM (short username kudu)
I20260928 02:13:59.110024 20332 webserver.cc:460] Webserver started at http://127.6.235.67:36033/ using document root <none> and password file <none>
I20260928 02:13:59.110868 20332 fs_manager.cc:359] Metadata directory not provided
I20260928 02:13:59.111045 20332 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 28 02:13:59 dist-test-slave-86gv krb5kdc[18975](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561639, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.235.67@KRBTEST.COM for HTTP/127.6.235.84@KRBTEST.COM
I20260928 02:13:59.137630 20332 fs_manager.cc:709] Time spent opening directory manager: real 0.006s user 0.005s sys 0.000s
I20260928 02:13:59.140810 20348 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:13:59.142724 20332 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260928 02:13:59.142943 20332 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790561432251112-7085-0/minicluster-data/ts-2/data,/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790561432251112-7085-0/minicluster-data/ts-2/wal
uuid: "ea5564b49bbf435bbb8a8008bf6b2643"
format_stamp: "Formatted at 2026-09-28 02:13:55 on dist-test-slave-86gv"
server_key: "1a83de9c040658ccc66e4181d8cd847a"
server_key_iv: "de93ba23b8d1a36fed968ca0bb21ca15"
server_key_version: "kuduclusterkey@0"
I20260928 02:13:59.143985 20332 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790561432251112-7085-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790561432251112-7085-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790561432251112-7085-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:13:59.164882 20332 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:13:59.169428 20332 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:13:59.169757 20332 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:13:59.170895 20332 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:13:59.172571 20332 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:13:59.172688 20332 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:13:59.172820 20332 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:13:59.173004 20332 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.001s
I20260928 02:13:59.222172 20332 rpc_server.cc:307] RPC server started. Bound to: 127.6.235.67:43127
I20260928 02:13:59.222168 20460 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.235.67:43127 every 8 connection(s)
I20260928 02:13:59.224226 20332 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790561432251112-7085-0/minicluster-data/ts-2/data/info.pb
I20260928 02:13:59.233068 7085 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTlaUuc/build/asan/bin/kudu as pid 20332
I20260928 02:13:59.271270 20464 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:13:59.234316 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54526 (local address 127.6.235.126:41799)
0928 02:13:59.251890 (+ 17574us) server_negotiation.cc:206] Beginning negotiation
0928 02:13:59.251897 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 02:13:59.251924 (+ 27us) server_negotiation.cc:407] Connection header received
0928 02:13:59.252017 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:13:59.252023 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:13:59.252354 (+ 331us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:13:59.253095 (+ 741us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:13:59.255167 (+ 2072us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:13:59.256418 (+ 1251us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:13:59.259584 (+ 3166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:13:59.260157 (+ 573us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:13:59.263602 (+ 3445us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:13:59.263862 (+ 260us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:13:59.263867 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:13:59.263871 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:13:59.263903 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:13:59.267242 (+ 3339us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:13:59.268174 (+ 932us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:13:59.268179 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:13:59.268182 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:13:59.268329 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:13:59.268773 (+ 444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:13:59.268782 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:13:59.268785 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:13:59.269179 (+ 394us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:13:59.270567 (+ 1388us) server_negotiation.cc:1035] Waiting for connection context
0928 02:13:59.270700 (+ 133us) server_negotiation.cc:299] Negotiation successful
0928 02:13:59.270923 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16348,"thread_start_us":14846,"threads_started":1}
Sep 28 02:13:59 dist-test-slave-86gv krb5kdc[18975](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561639, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.235.67@KRBTEST.COM for kudu/127.6.235.126@KRBTEST.COM
I20260928 02:13:59.293767 20465 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:13:59.228625 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.235.67:40573 (local address 127.6.235.126:41799)
0928 02:13:59.249807 (+ 21182us) server_negotiation.cc:206] Beginning negotiation
0928 02:13:59.249814 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 02:13:59.256658 (+ 6844us) server_negotiation.cc:407] Connection header received
0928 02:13:59.257446 (+ 788us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:13:59.257452 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:13:59.257542 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:13:59.257655 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:13:59.268733 (+ 11078us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:13:59.270060 (+ 1327us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:13:59.276815 (+ 6755us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:13:59.277559 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:13:59.277714 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:13:59.283193 (+ 5479us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:13:59.283201 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:13:59.283206 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:13:59.283267 (+ 61us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:13:59.287511 (+ 4244us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:13:59.290684 (+ 3173us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:13:59.290689 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:13:59.290693 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:13:59.290849 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:13:59.291438 (+ 589us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:13:59.291443 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:13:59.291446 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:13:59.291826 (+ 380us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:13:59.291961 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0928 02:13:59.293019 (+ 1058us) server_negotiation.cc:299] Negotiation successful
0928 02:13:59.293395 (+ 376us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20659,"thread_start_us":21891,"threads_started":1}
I20260928 02:13:59.295436 20463 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:13:59.228068 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.235.126:41799 (local address 127.6.235.67:40573)
0928 02:13:59.253879 (+ 25811us) negotiation.cc:107] Waiting for socket to connect
0928 02:13:59.253950 (+ 71us) client_negotiation.cc:174] Beginning negotiation
0928 02:13:59.257018 (+ 3068us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:13:59.260305 (+ 3287us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:13:59.260321 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:13:59.261006 (+ 685us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:13:59.268443 (+ 7437us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:13:59.268478 (+ 35us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:13:59.273959 (+ 5481us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:13:59.273966 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:13:59.276468 (+ 2502us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:13:59.276633 (+ 165us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:13:59.277943 (+ 1310us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:13:59.277953 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:13:59.278308 (+ 355us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:13:59.279316 (+ 1008us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:13:59.279345 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:13:59.282088 (+ 2743us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:13:59.289718 (+ 7630us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:13:59.289728 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:13:59.289749 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:13:59.290527 (+ 778us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:13:59.291006 (+ 479us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:13:59.291011 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:13:59.291014 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:13:59.291281 (+ 267us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:13:59.291990 (+ 709us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:13:59.291998 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:13:59.292482 (+ 484us) client_negotiation.cc:769] Sending connection context
0928 02:13:59.292838 (+ 356us) client_negotiation.cc:240] Negotiation successful
0928 02:13:59.293507 (+ 669us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":25449,"thread_start_us":25228,"threads_started":1}
I20260928 02:13:59.297227 20461 heartbeater.cc:347] Connected to a master server at 127.6.235.126:41799
I20260928 02:13:59.297566 20461 heartbeater.cc:464] Registering TS with master...
I20260928 02:13:59.298357 20461 heartbeater.cc:511] Master 127.6.235.126:41799 requested a full tablet report, sending...
Sep 28 02:13:59 dist-test-slave-86gv krb5kdc[18975](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561638, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.235.126@KRBTEST.COM for HTTP/127.6.235.84@KRBTEST.COM
I20260928 02:13:59.300004 19986 ts_manager.cc:194] Registered new tserver with Master: ea5564b49bbf435bbb8a8008bf6b2643 (127.6.235.67:43127)
I20260928 02:13:59.301173 19986 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.235.67@KRBTEST.COM'} at 127.6.235.67:40573
I20260928 02:13:59.303973 7085 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 02:13:59.328058 20465 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:13:59.309131 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54532 (local address 127.6.235.126:41799)
0928 02:13:59.309299 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0928 02:13:59.309305 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 02:13:59.311257 (+ 1952us) server_negotiation.cc:407] Connection header received
0928 02:13:59.311462 (+ 205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:13:59.311467 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:13:59.311562 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:13:59.311685 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:13:59.313755 (+ 2070us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:13:59.314829 (+ 1074us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:13:59.316760 (+ 1931us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:13:59.317433 (+ 673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:13:59.317579 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:13:59.320784 (+ 3205us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:13:59.320791 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:13:59.320796 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:13:59.320842 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:13:59.324539 (+ 3697us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:13:59.325518 (+ 979us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:13:59.325523 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:13:59.325527 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:13:59.325683 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:13:59.326213 (+ 530us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:13:59.326218 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:13:59.326221 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:13:59.326609 (+ 388us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:13:59.326847 (+ 238us) server_negotiation.cc:1035] Waiting for connection context
0928 02:13:59.327440 (+ 593us) server_negotiation.cc:299] Negotiation successful
0928 02:13:59.327720 (+ 280us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260928 02:13:59.333606 19985 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54532:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 02:13:59.344244 19985 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 02:13:59.450198 20479 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:13:59.414682 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.235.67:43127 (local address 127.0.0.1:49790)
0928 02:13:59.438981 (+ 24299us) negotiation.cc:107] Waiting for socket to connect
0928 02:13:59.439000 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0928 02:13:59.439108 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:13:59.442003 (+ 2895us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:13:59.442012 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:13:59.442045 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:13:59.442439 (+ 394us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:13:59.442451 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:13:59.444590 (+ 2139us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:13:59.444596 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:13:59.446893 (+ 2297us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:13:59.446909 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:13:59.449257 (+ 2348us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:13:59.449263 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:13:59.449460 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:13:59.449511 (+ 51us) client_negotiation.cc:769] Sending connection context
0928 02:13:59.449657 (+ 146us) client_negotiation.cc:240] Negotiation successful
0928 02:13:59.449847 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":24146,"thread_start_us":24002,"threads_started":1}
I20260928 02:13:59.457338 20480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:13:59.414932 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49790 (local address 127.6.235.67:43127)
0928 02:13:59.441052 (+ 26120us) server_negotiation.cc:206] Beginning negotiation
0928 02:13:59.441060 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 02:13:59.441094 (+ 34us) server_negotiation.cc:407] Connection header received
0928 02:13:59.441193 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:13:59.441201 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:13:59.441465 (+ 264us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:13:59.441688 (+ 223us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:13:59.442663 (+ 975us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:13:59.444230 (+ 1567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:13:59.447100 (+ 2870us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:13:59.448633 (+ 1533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:13:59.448833 (+ 200us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:13:59.449098 (+ 265us) server_negotiation.cc:1035] Waiting for connection context
0928 02:13:59.456745 (+ 7647us) server_negotiation.cc:299] Negotiation successful
0928 02:13:59.456984 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25904,"thread_start_us":29375,"threads_started":1}
I20260928 02:13:59.461679 20396 tablet_service.cc:1467] Processing CreateTablet for tablet ff9ab0a50ff54d8488e70f18dbf38578 (DEFAULT_TABLE table=test-table [id=d3843da5139d43488d2e5c7e672b2bd5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:13:59.463300 20396 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ff9ab0a50ff54d8488e70f18dbf38578. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:13:59.466687 20482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:13:59.403337 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.235.65:36397 (local address 127.0.0.1:40666)
0928 02:13:59.437746 (+ 34409us) negotiation.cc:107] Waiting for socket to connect
0928 02:13:59.437796 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0928 02:13:59.438007 (+ 211us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:13:59.454101 (+ 16094us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:13:59.454106 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:13:59.454132 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:13:59.454435 (+ 303us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:13:59.454446 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:13:59.457067 (+ 2621us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:13:59.457073 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:13:59.459298 (+ 2225us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:13:59.459315 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:13:59.465798 (+ 6483us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:13:59.465808 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:13:59.466017 (+ 209us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:13:59.466073 (+ 56us) client_negotiation.cc:769] Sending connection context
0928 02:13:59.466153 (+ 80us) client_negotiation.cc:240] Negotiation successful
0928 02:13:59.466308 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":34072,"thread_start_us":50423,"threads_started":1}
I20260928 02:13:59.474380 20478 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:13:59.403885 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40666 (local address 127.6.235.65:36397)
0928 02:13:59.448142 (+ 44257us) server_negotiation.cc:206] Beginning negotiation
0928 02:13:59.448153 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 02:13:59.448187 (+ 34us) server_negotiation.cc:407] Connection header received
0928 02:13:59.448294 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:13:59.448302 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:13:59.448872 (+ 570us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:13:59.453554 (+ 4682us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:13:59.454691 (+ 1137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:13:59.456333 (+ 1642us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:13:59.460133 (+ 3800us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:13:59.465515 (+ 5382us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:13:59.473313 (+ 7798us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:13:59.473618 (+ 305us) server_negotiation.cc:1035] Waiting for connection context
0928 02:13:59.473721 (+ 103us) server_negotiation.cc:299] Negotiation successful
0928 02:13:59.473973 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44049,"thread_start_us":44724,"threads_started":1}
I20260928 02:13:59.476327 20481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:13:59.421500 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55062 (local address 127.6.235.66:44645)
0928 02:13:59.440401 (+ 18901us) server_negotiation.cc:206] Beginning negotiation
0928 02:13:59.440410 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 02:13:59.450679 (+ 10269us) server_negotiation.cc:407] Connection header received
0928 02:13:59.451273 (+ 594us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:13:59.451282 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:13:59.451557 (+ 275us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:13:59.451768 (+ 211us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:13:59.452401 (+ 633us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:13:59.453955 (+ 1554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:13:59.473224 (+ 19269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:13:59.474806 (+ 1582us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:13:59.474932 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:13:59.475139 (+ 207us) server_negotiation.cc:1035] Waiting for connection context
0928 02:13:59.475813 (+ 674us) server_negotiation.cc:299] Negotiation successful
0928 02:13:59.476021 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18729,"thread_start_us":18911,"threads_started":1}
I20260928 02:13:59.477243 20479 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:13:59.421432 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.235.66:44645 (local address 127.0.0.1:55062)
0928 02:13:59.450562 (+ 29130us) negotiation.cc:107] Waiting for socket to connect
0928 02:13:59.450582 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0928 02:13:59.451061 (+ 479us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:13:59.452043 (+ 982us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:13:59.452049 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:13:59.452074 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:13:59.452235 (+ 161us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:13:59.452245 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:13:59.454655 (+ 2410us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:13:59.454660 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:13:59.456487 (+ 1827us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:13:59.456512 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:13:59.475339 (+ 18827us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:13:59.475347 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:13:59.475555 (+ 208us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:13:59.475593 (+ 38us) client_negotiation.cc:769] Sending connection context
0928 02:13:59.476711 (+ 1118us) client_negotiation.cc:240] Negotiation successful
0928 02:13:59.476936 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":29017,"thread_start_us":15824,"threads_started":1}
I20260928 02:13:59.481070 20118 tablet_service.cc:1467] Processing CreateTablet for tablet ff9ab0a50ff54d8488e70f18dbf38578 (DEFAULT_TABLE table=test-table [id=d3843da5139d43488d2e5c7e672b2bd5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:13:59.481561 20262 tablet_service.cc:1467] Processing CreateTablet for tablet ff9ab0a50ff54d8488e70f18dbf38578 (DEFAULT_TABLE table=test-table [id=d3843da5139d43488d2e5c7e672b2bd5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:13:59.482954 20118 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ff9ab0a50ff54d8488e70f18dbf38578. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:13:59.482955 20262 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ff9ab0a50ff54d8488e70f18dbf38578. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:13:59.513743 20483 tablet_bootstrap.cc:492] T ff9ab0a50ff54d8488e70f18dbf38578 P ea5564b49bbf435bbb8a8008bf6b2643: Bootstrap starting.
I20260928 02:13:59.515816 20485 tablet_bootstrap.cc:492] T ff9ab0a50ff54d8488e70f18dbf38578 P 269968aeb430404a8977927db1ba1660: Bootstrap starting.
I20260928 02:13:59.518610 20483 tablet_bootstrap.cc:654] T ff9ab0a50ff54d8488e70f18dbf38578 P ea5564b49bbf435bbb8a8008bf6b2643: Neither blocks nor log segments found. Creating new log.
I20260928 02:13:59.520589 20483 log.cc:824] T ff9ab0a50ff54d8488e70f18dbf38578 P ea5564b49bbf435bbb8a8008bf6b2643: Log is configured to *not* fsync() on all Append() calls
I20260928 02:13:59.521600 20484 tablet_bootstrap.cc:492] T ff9ab0a50ff54d8488e70f18dbf38578 P 290fd539aafb4fceabbcd6bf66c4b1bc: Bootstrap starting.
I20260928 02:13:59.525658 20484 tablet_bootstrap.cc:654] T ff9ab0a50ff54d8488e70f18dbf38578 P 290fd539aafb4fceabbcd6bf66c4b1bc: Neither blocks nor log segments found. Creating new log.
I20260928 02:13:59.527171 20484 log.cc:824] T ff9ab0a50ff54d8488e70f18dbf38578 P 290fd539aafb4fceabbcd6bf66c4b1bc: Log is configured to *not* fsync() on all Append() calls
I20260928 02:13:59.528870 20485 tablet_bootstrap.cc:654] T ff9ab0a50ff54d8488e70f18dbf38578 P 269968aeb430404a8977927db1ba1660: Neither blocks nor log segments found. Creating new log.
I20260928 02:13:59.530323 20485 log.cc:824] T ff9ab0a50ff54d8488e70f18dbf38578 P 269968aeb430404a8977927db1ba1660: Log is configured to *not* fsync() on all Append() calls
I20260928 02:13:59.536652 20483 tablet_bootstrap.cc:492] T ff9ab0a50ff54d8488e70f18dbf38578 P ea5564b49bbf435bbb8a8008bf6b2643: No bootstrap required, opened a new log
I20260928 02:13:59.537117 20483 ts_tablet_manager.cc:1397] T ff9ab0a50ff54d8488e70f18dbf38578 P ea5564b49bbf435bbb8a8008bf6b2643: Time spent bootstrapping tablet: real 0.024s user 0.011s sys 0.000s
I20260928 02:13:59.550710 20483 raft_consensus.cc:348] T ff9ab0a50ff54d8488e70f18dbf38578 P ea5564b49bbf435bbb8a8008bf6b2643 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "269968aeb430404a8977927db1ba1660" member_type: VOTER last_known_addr { host: "127.6.235.65" port: 36397 } } peers { permanent_uuid: "ea5564b49bbf435bbb8a8008bf6b2643" member_type: VOTER last_known_addr { host: "127.6.235.67" port: 43127 } } peers { permanent_uuid: "290fd539aafb4fceabbcd6bf66c4b1bc" member_type: VOTER last_known_addr { host: "127.6.235.66" port: 44645 } }
I20260928 02:13:59.556162 20485 tablet_bootstrap.cc:492] T ff9ab0a50ff54d8488e70f18dbf38578 P 269968aeb430404a8977927db1ba1660: No bootstrap required, opened a new log
I20260928 02:13:59.556625 20485 ts_tablet_manager.cc:1397] T ff9ab0a50ff54d8488e70f18dbf38578 P 269968aeb430404a8977927db1ba1660: Time spent bootstrapping tablet: real 0.041s user 0.005s sys 0.005s
I20260928 02:13:59.561167 20483 raft_consensus.cc:374] T ff9ab0a50ff54d8488e70f18dbf38578 P ea5564b49bbf435bbb8a8008bf6b2643 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:13:59.561416 20483 raft_consensus.cc:729] T ff9ab0a50ff54d8488e70f18dbf38578 P ea5564b49bbf435bbb8a8008bf6b2643 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ea5564b49bbf435bbb8a8008bf6b2643, State: Initialized, Role: FOLLOWER
I20260928 02:13:59.563534 20483 consensus_queue.cc:260] T ff9ab0a50ff54d8488e70f18dbf38578 P ea5564b49bbf435bbb8a8008bf6b2643 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "269968aeb430404a8977927db1ba1660" member_type: VOTER last_known_addr { host: "127.6.235.65" port: 36397 } } peers { permanent_uuid: "ea5564b49bbf435bbb8a8008bf6b2643" member_type: VOTER last_known_addr { host: "127.6.235.67" port: 43127 } } peers { permanent_uuid: "290fd539aafb4fceabbcd6bf66c4b1bc" member_type: VOTER last_known_addr { host: "127.6.235.66" port: 44645 } }
I20260928 02:13:59.569128 20485 raft_consensus.cc:348] T ff9ab0a50ff54d8488e70f18dbf38578 P 269968aeb430404a8977927db1ba1660 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "269968aeb430404a8977927db1ba1660" member_type: VOTER last_known_addr { host: "127.6.235.65" port: 36397 } } peers { permanent_uuid: "ea5564b49bbf435bbb8a8008bf6b2643" member_type: VOTER last_known_addr { host: "127.6.235.67" port: 43127 } } peers { permanent_uuid: "290fd539aafb4fceabbcd6bf66c4b1bc" member_type: VOTER last_known_addr { host: "127.6.235.66" port: 44645 } }
I20260928 02:13:59.571897 20484 tablet_bootstrap.cc:492] T ff9ab0a50ff54d8488e70f18dbf38578 P 290fd539aafb4fceabbcd6bf66c4b1bc: No bootstrap required, opened a new log
I20260928 02:13:59.572314 20484 ts_tablet_manager.cc:1397] T ff9ab0a50ff54d8488e70f18dbf38578 P 290fd539aafb4fceabbcd6bf66c4b1bc: Time spent bootstrapping tablet: real 0.051s user 0.020s sys 0.024s
I20260928 02:13:59.572372 20485 raft_consensus.cc:374] T ff9ab0a50ff54d8488e70f18dbf38578 P 269968aeb430404a8977927db1ba1660 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:13:59.572546 20485 raft_consensus.cc:729] T ff9ab0a50ff54d8488e70f18dbf38578 P 269968aeb430404a8977927db1ba1660 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 269968aeb430404a8977927db1ba1660, State: Initialized, Role: FOLLOWER
I20260928 02:13:59.573472 20485 consensus_queue.cc:260] T ff9ab0a50ff54d8488e70f18dbf38578 P 269968aeb430404a8977927db1ba1660 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "269968aeb430404a8977927db1ba1660" member_type: VOTER last_known_addr { host: "127.6.235.65" port: 36397 } } peers { permanent_uuid: "ea5564b49bbf435bbb8a8008bf6b2643" member_type: VOTER last_known_addr { host: "127.6.235.67" port: 43127 } } peers { permanent_uuid: "290fd539aafb4fceabbcd6bf66c4b1bc" member_type: VOTER last_known_addr { host: "127.6.235.66" port: 44645 } }
I20260928 02:13:59.582242 20461 heartbeater.cc:503] Master 127.6.235.126:41799 was elected leader, sending a full tablet report...
I20260928 02:13:59.585637 20183 heartbeater.cc:503] Master 127.6.235.126:41799 was elected leader, sending a full tablet report...
I20260928 02:13:59.586489 20485 ts_tablet_manager.cc:1428] T ff9ab0a50ff54d8488e70f18dbf38578 P 269968aeb430404a8977927db1ba1660: Time spent starting tablet: real 0.030s user 0.014s sys 0.010s
I20260928 02:13:59.587675 20490 raft_consensus.cc:482] T ff9ab0a50ff54d8488e70f18dbf38578 P ea5564b49bbf435bbb8a8008bf6b2643 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 02:13:59.588052 20490 raft_consensus.cc:504] T ff9ab0a50ff54d8488e70f18dbf38578 P ea5564b49bbf435bbb8a8008bf6b2643 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "269968aeb430404a8977927db1ba1660" member_type: VOTER last_known_addr { host: "127.6.235.65" port: 36397 } } peers { permanent_uuid: "ea5564b49bbf435bbb8a8008bf6b2643" member_type: VOTER last_known_addr { host: "127.6.235.67" port: 43127 } } peers { permanent_uuid: "290fd539aafb4fceabbcd6bf66c4b1bc" member_type: VOTER last_known_addr { host: "127.6.235.66" port: 44645 } }
I20260928 02:13:59.588886 20484 raft_consensus.cc:348] T ff9ab0a50ff54d8488e70f18dbf38578 P 290fd539aafb4fceabbcd6bf66c4b1bc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "269968aeb430404a8977927db1ba1660" member_type: VOTER last_known_addr { host: "127.6.235.65" port: 36397 } } peers { permanent_uuid: "ea5564b49bbf435bbb8a8008bf6b2643" member_type: VOTER last_known_addr { host: "127.6.235.67" port: 43127 } } peers { permanent_uuid: "290fd539aafb4fceabbcd6bf66c4b1bc" member_type: VOTER last_known_addr { host: "127.6.235.66" port: 44645 } }
I20260928 02:13:59.589471 20484 raft_consensus.cc:374] T ff9ab0a50ff54d8488e70f18dbf38578 P 290fd539aafb4fceabbcd6bf66c4b1bc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:13:59.589630 20484 raft_consensus.cc:729] T ff9ab0a50ff54d8488e70f18dbf38578 P 290fd539aafb4fceabbcd6bf66c4b1bc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 290fd539aafb4fceabbcd6bf66c4b1bc, State: Initialized, Role: FOLLOWER
I20260928 02:13:59.590468 20484 consensus_queue.cc:260] T ff9ab0a50ff54d8488e70f18dbf38578 P 290fd539aafb4fceabbcd6bf66c4b1bc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "269968aeb430404a8977927db1ba1660" member_type: VOTER last_known_addr { host: "127.6.235.65" port: 36397 } } peers { permanent_uuid: "ea5564b49bbf435bbb8a8008bf6b2643" member_type: VOTER last_known_addr { host: "127.6.235.67" port: 43127 } } peers { permanent_uuid: "290fd539aafb4fceabbcd6bf66c4b1bc" member_type: VOTER last_known_addr { host: "127.6.235.66" port: 44645 } }
I20260928 02:13:59.592146 20328 heartbeater.cc:503] Master 127.6.235.126:41799 was elected leader, sending a full tablet report...
I20260928 02:13:59.592967 20484 ts_tablet_manager.cc:1428] T ff9ab0a50ff54d8488e70f18dbf38578 P 290fd539aafb4fceabbcd6bf66c4b1bc: Time spent starting tablet: real 0.020s user 0.015s sys 0.003s
I20260928 02:13:59.599967 20463 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:13:59.590534 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.235.65:36397 (local address 127.6.235.67:33803)
0928 02:13:59.590907 (+ 373us) negotiation.cc:107] Waiting for socket to connect
0928 02:13:59.590929 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0928 02:13:59.591082 (+ 153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:13:59.593566 (+ 2484us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:13:59.593572 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:13:59.593605 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:13:59.593799 (+ 194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:13:59.593812 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:13:59.595261 (+ 1449us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:13:59.595267 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:13:59.597275 (+ 2008us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:13:59.597293 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:13:59.599144 (+ 1851us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:13:59.599160 (+ 16us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:13:59.599358 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:13:59.599426 (+ 68us) client_negotiation.cc:769] Sending connection context
0928 02:13:59.599538 (+ 112us) client_negotiation.cc:240] Negotiation successful
0928 02:13:59.599685 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":249}
I20260928 02:13:59.600735 20478 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:13:59.591403 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.235.67:33803 (local address 127.6.235.65:36397)
0928 02:13:59.591895 (+ 492us) server_negotiation.cc:206] Beginning negotiation
0928 02:13:59.591901 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 02:13:59.591921 (+ 20us) server_negotiation.cc:407] Connection header received
0928 02:13:59.591986 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:13:59.591992 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:13:59.592093 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:13:59.592192 (+ 99us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:13:59.593988 (+ 1796us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:13:59.595090 (+ 1102us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:13:59.597471 (+ 2381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:13:59.598699 (+ 1228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:13:59.598863 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:13:59.598942 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0928 02:13:59.600306 (+ 1364us) server_negotiation.cc:299] Negotiation successful
0928 02:13:59.600468 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":351}
I20260928 02:13:59.602108 20138 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff9ab0a50ff54d8488e70f18dbf38578" candidate_uuid: "ea5564b49bbf435bbb8a8008bf6b2643" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "269968aeb430404a8977927db1ba1660" is_pre_election: true
I20260928 02:13:59.602767 20138 raft_consensus.cc:2454] T ff9ab0a50ff54d8488e70f18dbf38578 P 269968aeb430404a8977927db1ba1660 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ea5564b49bbf435bbb8a8008bf6b2643 in term 0.
I20260928 02:13:59.603701 20350 leader_election.cc:302] T ff9ab0a50ff54d8488e70f18dbf38578 P ea5564b49bbf435bbb8a8008bf6b2643 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 269968aeb430404a8977927db1ba1660, ea5564b49bbf435bbb8a8008bf6b2643; no voters:
I20260928 02:13:59.604313 20489 raft_consensus.cc:2781] T ff9ab0a50ff54d8488e70f18dbf38578 P ea5564b49bbf435bbb8a8008bf6b2643 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 02:13:59.604552 20489 raft_consensus.cc:482] T ff9ab0a50ff54d8488e70f18dbf38578 P ea5564b49bbf435bbb8a8008bf6b2643 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 02:13:59.604751 20489 raft_consensus.cc:3037] T ff9ab0a50ff54d8488e70f18dbf38578 P ea5564b49bbf435bbb8a8008bf6b2643 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:13:59.608323 20489 raft_consensus.cc:504] T ff9ab0a50ff54d8488e70f18dbf38578 P ea5564b49bbf435bbb8a8008bf6b2643 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "269968aeb430404a8977927db1ba1660" member_type: VOTER last_known_addr { host: "127.6.235.65" port: 36397 } } peers { permanent_uuid: "ea5564b49bbf435bbb8a8008bf6b2643" member_type: VOTER last_known_addr { host: "127.6.235.67" port: 43127 } } peers { permanent_uuid: "290fd539aafb4fceabbcd6bf66c4b1bc" member_type: VOTER last_known_addr { host: "127.6.235.66" port: 44645 } }
I20260928 02:13:59.609639 20138 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff9ab0a50ff54d8488e70f18dbf38578" candidate_uuid: "ea5564b49bbf435bbb8a8008bf6b2643" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "269968aeb430404a8977927db1ba1660"
I20260928 02:13:59.609866 20138 raft_consensus.cc:3037] T ff9ab0a50ff54d8488e70f18dbf38578 P 269968aeb430404a8977927db1ba1660 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:13:59.610087 20481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:13:59.602141 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.235.67:48975 (local address 127.6.235.66:44645)
0928 02:13:59.602356 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0928 02:13:59.602363 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 02:13:59.603006 (+ 643us) server_negotiation.cc:407] Connection header received
0928 02:13:59.603269 (+ 263us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:13:59.603274 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:13:59.603386 (+ 112us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:13:59.603463 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:13:59.604056 (+ 593us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:13:59.605220 (+ 1164us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:13:59.607562 (+ 2342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:13:59.608866 (+ 1304us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:13:59.609019 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:13:59.609096 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0928 02:13:59.609692 (+ 596us) server_negotiation.cc:299] Negotiation successful
0928 02:13:59.609821 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260928 02:13:59.610739 20463 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:13:59.602631 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.235.66:44645 (local address 127.6.235.67:48975)
0928 02:13:59.602919 (+ 288us) negotiation.cc:107] Waiting for socket to connect
0928 02:13:59.602940 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0928 02:13:59.603113 (+ 173us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:13:59.603650 (+ 537us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:13:59.603656 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:13:59.603693 (+ 37us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:13:59.603895 (+ 202us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:13:59.603909 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:13:59.605442 (+ 1533us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:13:59.605449 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:13:59.607386 (+ 1937us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:13:59.607408 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:13:59.609254 (+ 1846us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:13:59.609260 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:13:59.609485 (+ 225us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:13:59.609519 (+ 34us) client_negotiation.cc:769] Sending connection context
0928 02:13:59.610316 (+ 797us) client_negotiation.cc:240] Negotiation successful
0928 02:13:59.610481 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":174}
I20260928 02:13:59.611876 20283 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff9ab0a50ff54d8488e70f18dbf38578" candidate_uuid: "ea5564b49bbf435bbb8a8008bf6b2643" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "290fd539aafb4fceabbcd6bf66c4b1bc" is_pre_election: true
I20260928 02:13:59.612504 20283 raft_consensus.cc:2454] T ff9ab0a50ff54d8488e70f18dbf38578 P 290fd539aafb4fceabbcd6bf66c4b1bc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ea5564b49bbf435bbb8a8008bf6b2643 in term 0.
I20260928 02:13:59.612759 20483 ts_tablet_manager.cc:1428] T ff9ab0a50ff54d8488e70f18dbf38578 P ea5564b49bbf435bbb8a8008bf6b2643: Time spent starting tablet: real 0.076s user 0.013s sys 0.007s
I20260928 02:13:59.613163 20138 raft_consensus.cc:2454] T ff9ab0a50ff54d8488e70f18dbf38578 P 269968aeb430404a8977927db1ba1660 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ea5564b49bbf435bbb8a8008bf6b2643 in term 1.
I20260928 02:13:59.613718 20490 leader_election.cc:288] T ff9ab0a50ff54d8488e70f18dbf38578 P ea5564b49bbf435bbb8a8008bf6b2643 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 269968aeb430404a8977927db1ba1660 (127.6.235.65:36397), 290fd539aafb4fceabbcd6bf66c4b1bc (127.6.235.66:44645)
I20260928 02:13:59.614032 20350 leader_election.cc:302] T ff9ab0a50ff54d8488e70f18dbf38578 P ea5564b49bbf435bbb8a8008bf6b2643 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 269968aeb430404a8977927db1ba1660, ea5564b49bbf435bbb8a8008bf6b2643; no voters:
W20260928 02:13:59.614694 20329 tablet.cc:2369] T ff9ab0a50ff54d8488e70f18dbf38578 P 290fd539aafb4fceabbcd6bf66c4b1bc: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 02:13:59.614382 20490 raft_consensus.cc:2781] T ff9ab0a50ff54d8488e70f18dbf38578 P ea5564b49bbf435bbb8a8008bf6b2643 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:13:59.618120 20489 leader_election.cc:288] T ff9ab0a50ff54d8488e70f18dbf38578 P ea5564b49bbf435bbb8a8008bf6b2643 [CANDIDATE]: Term 1 election: Requested vote from peers 269968aeb430404a8977927db1ba1660 (127.6.235.65:36397), 290fd539aafb4fceabbcd6bf66c4b1bc (127.6.235.66:44645)
I20260928 02:13:59.618665 20283 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff9ab0a50ff54d8488e70f18dbf38578" candidate_uuid: "ea5564b49bbf435bbb8a8008bf6b2643" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "290fd539aafb4fceabbcd6bf66c4b1bc"
I20260928 02:13:59.618916 20283 raft_consensus.cc:3037] T ff9ab0a50ff54d8488e70f18dbf38578 P 290fd539aafb4fceabbcd6bf66c4b1bc [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:13:59.622447 20283 raft_consensus.cc:2454] T ff9ab0a50ff54d8488e70f18dbf38578 P 290fd539aafb4fceabbcd6bf66c4b1bc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ea5564b49bbf435bbb8a8008bf6b2643 in term 1.
I20260928 02:13:59.622463 20490 raft_consensus.cc:686] T ff9ab0a50ff54d8488e70f18dbf38578 P ea5564b49bbf435bbb8a8008bf6b2643 [term 1 LEADER]: Becoming Leader. State: Replica: ea5564b49bbf435bbb8a8008bf6b2643, State: Running, Role: LEADER
I20260928 02:13:59.623140 20490 consensus_queue.cc:237] T ff9ab0a50ff54d8488e70f18dbf38578 P ea5564b49bbf435bbb8a8008bf6b2643 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "269968aeb430404a8977927db1ba1660" member_type: VOTER last_known_addr { host: "127.6.235.65" port: 36397 } } peers { permanent_uuid: "ea5564b49bbf435bbb8a8008bf6b2643" member_type: VOTER last_known_addr { host: "127.6.235.67" port: 43127 } } peers { permanent_uuid: "290fd539aafb4fceabbcd6bf66c4b1bc" member_type: VOTER last_known_addr { host: "127.6.235.66" port: 44645 } }
I20260928 02:13:59.629992 19985 catalog_manager.cc:5495] T ff9ab0a50ff54d8488e70f18dbf38578 P ea5564b49bbf435bbb8a8008bf6b2643 reported cstate change: term changed from 0 to 1, leader changed from <none> to ea5564b49bbf435bbb8a8008bf6b2643 (127.6.235.67). New cstate: current_term: 1 leader_uuid: "ea5564b49bbf435bbb8a8008bf6b2643" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "269968aeb430404a8977927db1ba1660" member_type: VOTER last_known_addr { host: "127.6.235.65" port: 36397 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ea5564b49bbf435bbb8a8008bf6b2643" member_type: VOTER last_known_addr { host: "127.6.235.67" port: 43127 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "290fd539aafb4fceabbcd6bf66c4b1bc" member_type: VOTER last_known_addr { host: "127.6.235.66" port: 44645 } health_report { overall_health: UNKNOWN } } }
I20260928 02:13:59.651226 20480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:13:59.645804 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49794 (local address 127.6.235.67:43127)
0928 02:13:59.645977 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0928 02:13:59.645981 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:13:59.646145 (+ 164us) server_negotiation.cc:407] Connection header received
0928 02:13:59.646277 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:13:59.646281 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:13:59.646358 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:13:59.646517 (+ 159us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 02:13:59.646847 (+ 330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:13:59.647715 (+ 868us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:13:59.649256 (+ 1541us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:13:59.649656 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:13:59.649750 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:13:59.650209 (+ 459us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 02:13:59.650738 (+ 529us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 02:13:59.650863 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0928 02:13:59.650944 (+ 81us) server_negotiation.cc:299] Negotiation successful
0928 02:13:59.651013 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260928 02:13:59.659513 20138 raft_consensus.cc:1261] T ff9ab0a50ff54d8488e70f18dbf38578 P 269968aeb430404a8977927db1ba1660 [term 1 FOLLOWER]: Refusing update from remote peer ea5564b49bbf435bbb8a8008bf6b2643: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 02:13:59.659610 20283 raft_consensus.cc:1261] T ff9ab0a50ff54d8488e70f18dbf38578 P 290fd539aafb4fceabbcd6bf66c4b1bc [term 1 FOLLOWER]: Refusing update from remote peer ea5564b49bbf435bbb8a8008bf6b2643: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 02:13:59.660682 20495 consensus_queue.cc:1035] T ff9ab0a50ff54d8488e70f18dbf38578 P ea5564b49bbf435bbb8a8008bf6b2643 [LEADER]: Connected to new peer: Peer: permanent_uuid: "269968aeb430404a8977927db1ba1660" member_type: VOTER last_known_addr { host: "127.6.235.65" port: 36397 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 02:13:59.661074 20490 consensus_queue.cc:1035] T ff9ab0a50ff54d8488e70f18dbf38578 P ea5564b49bbf435bbb8a8008bf6b2643 [LEADER]: Connected to new peer: Peer: permanent_uuid: "290fd539aafb4fceabbcd6bf66c4b1bc" member_type: VOTER last_known_addr { host: "127.6.235.66" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 02:13:59.671795 20503 mvcc.cc:205] Tried to move back new op lower bound from 7334140476035256320 to 7334140475904720896. Current Snapshot: MvccSnapshot[applied={T|T < 7334140476035256320}]
I20260928 02:13:59.672444 20504 mvcc.cc:205] Tried to move back new op lower bound from 7334140476035256320 to 7334140475904720896. Current Snapshot: MvccSnapshot[applied={T|T < 7334140476035256320}]
I20260928 02:13:59.690150 19986 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54532:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 02:13:59.690448 19986 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54532:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 02:13:59.694080 19986 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 5948defd0e5e4ca39ca7d4f2883db7eb: Sending DeleteTablet for 3 replicas of tablet ff9ab0a50ff54d8488e70f18dbf38578
I20260928 02:13:59.695276 20118 tablet_service.cc:1514] Processing DeleteTablet for tablet ff9ab0a50ff54d8488e70f18dbf38578 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:13:59 UTC) from {username='kudu', principal='kudu/127.6.235.126@KRBTEST.COM'} at 127.0.0.1:40666
I20260928 02:13:59.695307 20262 tablet_service.cc:1514] Processing DeleteTablet for tablet ff9ab0a50ff54d8488e70f18dbf38578 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:13:59 UTC) from {username='kudu', principal='kudu/127.6.235.126@KRBTEST.COM'} at 127.0.0.1:55062
I20260928 02:13:59.695317 20396 tablet_service.cc:1514] Processing DeleteTablet for tablet ff9ab0a50ff54d8488e70f18dbf38578 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:13:59 UTC) from {username='kudu', principal='kudu/127.6.235.126@KRBTEST.COM'} at 127.0.0.1:49790
I20260928 02:13:59.708549 20513 tablet_replica.cc:330] T ff9ab0a50ff54d8488e70f18dbf38578 P 290fd539aafb4fceabbcd6bf66c4b1bc: stopping tablet replica
I20260928 02:13:59.709149 20513 raft_consensus.cc:2229] T ff9ab0a50ff54d8488e70f18dbf38578 P 290fd539aafb4fceabbcd6bf66c4b1bc [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 02:13:59.709692 20513 raft_consensus.cc:2258] T ff9ab0a50ff54d8488e70f18dbf38578 P 290fd539aafb4fceabbcd6bf66c4b1bc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 02:13:59.709838 7085 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskTlaUuc/build/asan/bin/kudu pid 20023
I20260928 02:13:59.710505 20512 tablet_replica.cc:330] T ff9ab0a50ff54d8488e70f18dbf38578 P 269968aeb430404a8977927db1ba1660: stopping tablet replica
I20260928 02:13:59.710744 20514 tablet_replica.cc:330] T ff9ab0a50ff54d8488e70f18dbf38578 P ea5564b49bbf435bbb8a8008bf6b2643: stopping tablet replica
I20260928 02:13:59.710933 20512 raft_consensus.cc:2229] T ff9ab0a50ff54d8488e70f18dbf38578 P 269968aeb430404a8977927db1ba1660 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 02:13:59.711254 20514 raft_consensus.cc:2229] T ff9ab0a50ff54d8488e70f18dbf38578 P ea5564b49bbf435bbb8a8008bf6b2643 [term 1 LEADER]: Raft consensus shutting down.
I20260928 02:13:59.711427 20512 raft_consensus.cc:2258] T ff9ab0a50ff54d8488e70f18dbf38578 P 269968aeb430404a8977927db1ba1660 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 02:13:59.711949 20514 raft_consensus.cc:2258] T ff9ab0a50ff54d8488e70f18dbf38578 P ea5564b49bbf435bbb8a8008bf6b2643 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 02:13:59.714133 20514 ts_tablet_manager.cc:1905] T ff9ab0a50ff54d8488e70f18dbf38578 P ea5564b49bbf435bbb8a8008bf6b2643: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 02:13:59.715021 20513 ts_tablet_manager.cc:1905] T ff9ab0a50ff54d8488e70f18dbf38578 P 290fd539aafb4fceabbcd6bf66c4b1bc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 02:13:59.715687 20512 ts_tablet_manager.cc:1905] T ff9ab0a50ff54d8488e70f18dbf38578 P 269968aeb430404a8977927db1ba1660: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 02:13:59.721403 20514 ts_tablet_manager.cc:1918] T ff9ab0a50ff54d8488e70f18dbf38578 P ea5564b49bbf435bbb8a8008bf6b2643: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 02:13:59.721596 20514 log.cc:1196] T ff9ab0a50ff54d8488e70f18dbf38578 P ea5564b49bbf435bbb8a8008bf6b2643: Deleting WAL directory at /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790561432251112-7085-0/minicluster-data/ts-2/wal/wals/ff9ab0a50ff54d8488e70f18dbf38578
I20260928 02:13:59.722179 20514 ts_tablet_manager.cc:1939] T ff9ab0a50ff54d8488e70f18dbf38578 P ea5564b49bbf435bbb8a8008bf6b2643: Deleting consensus metadata
I20260928 02:13:59.722560 20513 ts_tablet_manager.cc:1918] T ff9ab0a50ff54d8488e70f18dbf38578 P 290fd539aafb4fceabbcd6bf66c4b1bc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 02:13:59.722812 20513 log.cc:1196] T ff9ab0a50ff54d8488e70f18dbf38578 P 290fd539aafb4fceabbcd6bf66c4b1bc: Deleting WAL directory at /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790561432251112-7085-0/minicluster-data/ts-1/wal/wals/ff9ab0a50ff54d8488e70f18dbf38578
I20260928 02:13:59.723395 20513 ts_tablet_manager.cc:1939] T ff9ab0a50ff54d8488e70f18dbf38578 P 290fd539aafb4fceabbcd6bf66c4b1bc: Deleting consensus metadata
I20260928 02:13:59.724229 19972 catalog_manager.cc:4841] TS ea5564b49bbf435bbb8a8008bf6b2643 (127.6.235.67:43127): tablet ff9ab0a50ff54d8488e70f18dbf38578 (table test-table [id=d3843da5139d43488d2e5c7e672b2bd5]) successfully deleted
I20260928 02:13:59.725558 19973 catalog_manager.cc:4841] TS 290fd539aafb4fceabbcd6bf66c4b1bc (127.6.235.66:44645): tablet ff9ab0a50ff54d8488e70f18dbf38578 (table test-table [id=d3843da5139d43488d2e5c7e672b2bd5]) successfully deleted
I20260928 02:13:59.728818 20512 ts_tablet_manager.cc:1918] T ff9ab0a50ff54d8488e70f18dbf38578 P 269968aeb430404a8977927db1ba1660: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 02:13:59.729048 20512 log.cc:1196] T ff9ab0a50ff54d8488e70f18dbf38578 P 269968aeb430404a8977927db1ba1660: Deleting WAL directory at /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790561432251112-7085-0/minicluster-data/ts-0/wal/wals/ff9ab0a50ff54d8488e70f18dbf38578
I20260928 02:13:59.729650 20512 ts_tablet_manager.cc:1939] T ff9ab0a50ff54d8488e70f18dbf38578 P 269968aeb430404a8977927db1ba1660: Deleting consensus metadata
I20260928 02:13:59.730434 20478 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:13:59.712194 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40682 (local address 127.6.235.65:36397)
0928 02:13:59.712863 (+ 669us) server_negotiation.cc:206] Beginning negotiation
0928 02:13:59.712870 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 02:13:59.712915 (+ 45us) server_negotiation.cc:407] Connection header received
0928 02:13:59.712983 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:13:59.712991 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:13:59.713093 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:13:59.713213 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:13:59.714917 (+ 1704us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:13:59.716017 (+ 1100us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:13:59.717963 (+ 1946us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:13:59.718670 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:13:59.718806 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:13:59.722191 (+ 3385us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:13:59.722213 (+ 22us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:13:59.722232 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:13:59.722275 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:13:59.726623 (+ 4348us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:13:59.727624 (+ 1001us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:13:59.727634 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:13:59.727638 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:13:59.727812 (+ 174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:13:59.728269 (+ 457us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:13:59.728274 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:13:59.728278 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:13:59.728787 (+ 509us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:13:59.729191 (+ 404us) server_negotiation.cc:1035] Waiting for connection context
0928 02:13:59.729736 (+ 545us) server_negotiation.cc:299] Negotiation successful
0928 02:13:59.729990 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":547}
I20260928 02:13:59.731200 20178 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 02:13:59.731666 19971 catalog_manager.cc:4841] TS 269968aeb430404a8977927db1ba1660 (127.6.235.65:36397): tablet ff9ab0a50ff54d8488e70f18dbf38578 (table test-table [id=d3843da5139d43488d2e5c7e672b2bd5]) successfully deleted
I20260928 02:13:59.868274 7085 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTlaUuc/build/asan/bin/kudu with pid 20023
I20260928 02:13:59.883844 7085 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskTlaUuc/build/asan/bin/kudu pid 20187
I20260928 02:13:59.899818 20481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:13:59.884733 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55068 (local address 127.6.235.66:44645)
0928 02:13:59.884939 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0928 02:13:59.884946 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 02:13:59.885204 (+ 258us) server_negotiation.cc:407] Connection header received
0928 02:13:59.885390 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:13:59.885394 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:13:59.885460 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:13:59.885561 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:13:59.887329 (+ 1768us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:13:59.888149 (+ 820us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:13:59.890057 (+ 1908us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:13:59.890446 (+ 389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:13:59.890561 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:13:59.893943 (+ 3382us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:13:59.893958 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:13:59.893975 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:13:59.894010 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:13:59.896932 (+ 2922us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:13:59.897840 (+ 908us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:13:59.897848 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:13:59.897851 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:13:59.897980 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:13:59.898381 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:13:59.898385 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:13:59.898387 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:13:59.898745 (+ 358us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:13:59.898953 (+ 208us) server_negotiation.cc:1035] Waiting for connection context
0928 02:13:59.899306 (+ 353us) server_negotiation.cc:299] Negotiation successful
0928 02:13:59.899467 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260928 02:13:59.900755 20323 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 02:14:00.008138 7085 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTlaUuc/build/asan/bin/kudu with pid 20187
I20260928 02:14:00.026089 7085 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskTlaUuc/build/asan/bin/kudu pid 20332
I20260928 02:14:00.041615 20480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:14:00.027048 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49796 (local address 127.6.235.67:43127)
0928 02:14:00.027269 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0928 02:14:00.027277 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 02:14:00.027301 (+ 24us) server_negotiation.cc:407] Connection header received
0928 02:14:00.027446 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:14:00.027452 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:14:00.027562 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:14:00.027707 (+ 145us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:14:00.029077 (+ 1370us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:14:00.029952 (+ 875us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:14:00.031379 (+ 1427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:14:00.031881 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:14:00.032040 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:14:00.034813 (+ 2773us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:14:00.034849 (+ 36us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:14:00.034867 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:14:00.034923 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:14:00.038249 (+ 3326us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:14:00.039063 (+ 814us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:14:00.039071 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:14:00.039074 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:14:00.039283 (+ 209us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:14:00.039733 (+ 450us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:14:00.039738 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:14:00.039742 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:14:00.040114 (+ 372us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:14:00.040462 (+ 348us) server_negotiation.cc:1035] Waiting for connection context
0928 02:14:00.040803 (+ 341us) server_negotiation.cc:299] Negotiation successful
0928 02:14:00.041070 (+ 267us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":93}
I20260928 02:14:00.042567 20456 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 02:14:00.167092 7085 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTlaUuc/build/asan/bin/kudu with pid 20332
I20260928 02:14:00.188767 7085 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskTlaUuc/build/asan/bin/kudu pid 19953
I20260928 02:14:00.189994 20016 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 02:14:00.305676 7085 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTlaUuc/build/asan/bin/kudu with pid 19953
I20260928 02:14:00.329694 7085 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260928 02:14:00.665355 7085 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260928 02:14:00.665623 7085 mini_ranger.cc:66] Stopping Ranger...
I20260928 02:14:01.011860 7085 mini_ranger.cc:68] Stopped Ranger
2026-09-28 02:14:01.012 UTC [19005] LOG: received smart shutdown request
2026-09-28 02:14:01.015 UTC [19005] LOG: background worker "logical replication launcher" (PID 19014) exited with exit code 1
2026-09-28 02:14:01.016 UTC [19009] LOG: shutting down
2026-09-28 02:14:01.218 UTC [19005] LOG: database system is shut down
2026-09-28T02:14:01Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers (54729 ms)
[ RUN ] SecurityITest.IPKICACert
Loading random data
Initializing database '/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790561432251112-7085-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 02:14:01 dist-test-slave-86gv krb5kdc[20527](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 02:14:01 dist-test-slave-86gv krb5kdc[20527](info): set up 2 sockets
Sep 28 02:14:01 dist-test-slave-86gv krb5kdc[20527](info): commencing operation
krb5kdc: starting...
W20260928 02:14:02.517793 7085 mini_kdc.cc:121] Time spent starting KDC: real 1.127s user 0.000s sys 0.026s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790561432251112-7085-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790561432251112-7085-0/krb5kdc/test-user.keytab.
Sep 28 02:14:02 dist-test-slave-86gv krb5kdc[20527](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561642, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T02:14:02Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T02:14:02Z Disabled control of system clock
WARNING: no policy specified for kudu/127.6.235.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.235.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.235.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790561432251112-7085-0/krb5kdc/kudu_127.6.235.126.keytab.
Entry for principal kudu/127.6.235.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790561432251112-7085-0/krb5kdc/kudu_127.6.235.126.keytab.
I20260928 02:14:02.768744 7085 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTlaUuc/build/asan/bin/kudu
/tmp/dist-test-taskTlaUuc/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790561432251112-7085-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790561432251112-7085-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790561432251112-7085-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790561432251112-7085-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.235.126:41361
--webserver_interface=127.6.235.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790561432251112-7085-0/krb5kdc/kudu_127.6.235.126.keytab
--principal=kudu/127.6.235.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.235.84:33969
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.235.126:41361,127.6.235.125:36985,127.6.235.124:45773
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790561432251112-7085-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790561432251112-7085-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790561432251112-7085-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:14:03.121665 20540 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:14:03.122035 20540 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:14:03.122128 20540 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:14:03.129405 20540 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 02:14:03.129519 20540 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:14:03.129565 20540 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:14:03.129628 20540 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 02:14:03.129673 20540 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 02:14:03.139686 20540 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.235.84:33969
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790561432251112-7085-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790561432251112-7085-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.235.126:41361,127.6.235.125:36985,127.6.235.124:45773
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790561432251112-7085-0/krb5kdc/kudu_127.6.235.126.keytab
--principal=kudu/127.6.235.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.235.126:41361
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790561432251112-7085-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.235.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790561432251112-7085-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 00:51:59 UTC on ccycloud.quasar-axgkwn.root.comops.site
build id 4676464
ASAN enabled
I20260928 02:14:03.141070 20540 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:14:03.142494 20540 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:14:03.155131 20545 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:14:03.155081 20546 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:14:03.155427 20548 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:14:03.156061 20540 server_base.cc:1034] running on GCE node
I20260928 02:14:03.156838 20540 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:14:03.158329 20540 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:14:03.159538 20540 hybrid_clock.cc:648] HybridClock initialized: now 1790561643159494 us; error 32 us; skew 500 ppm
I20260928 02:14:03.159838 20540 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 02:14:03 dist-test-slave-86gv krb5kdc[20527](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561643, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.235.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:14:03.163053 20540 init.cc:377] Logged in from keytab as kudu/127.6.235.126@KRBTEST.COM (short username kudu)
I20260928 02:14:03.164641 20540 webserver.cc:460] Webserver started at http://127.6.235.126:37745/ using document root <none> and password file <none>
I20260928 02:14:03.165450 20540 fs_manager.cc:359] Metadata directory not provided
I20260928 02:14:03.165556 20540 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:14:03.165822 20540 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:14:03.168838 20540 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790561432251112-7085-0/minicluster-data/master-0/data/instance:
uuid: "680b03dcbe2a4f85b1d399dd62507518"
format_stamp: "Formatted at 2026-09-28 02:14:03 on dist-test-slave-86gv"
server_key: "ba44e0c7754b60d38bed06a13a5ce97b"
server_key_iv: "86873a46c3c60507e0da43aea4ac8581"
server_key_version: "encryptionkey@0"
I20260928 02:14:03.169703 20540 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790561432251112-7085-0/minicluster-data/master-0/wal/instance:
uuid: "680b03dcbe2a4f85b1d399dd62507518"
format_stamp: "Formatted at 2026-09-28 02:14:03 on dist-test-slave-86gv"
server_key: "ba44e0c7754b60d38bed06a13a5ce97b"
server_key_iv: "86873a46c3c60507e0da43aea4ac8581"
server_key_version: "encryptionkey@0"
I20260928 02:14:03.174978 20540 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260928 02:14:03.179020 20555 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:14:03.180585 20540 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260928 02:14:03.180747 20540 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790561432251112-7085-0/minicluster-data/master-0/data,/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790561432251112-7085-0/minicluster-data/master-0/wal
uuid: "680b03dcbe2a4f85b1d399dd62507518"
format_stamp: "Formatted at 2026-09-28 02:14:03 on dist-test-slave-86gv"
server_key: "ba44e0c7754b60d38bed06a13a5ce97b"
server_key_iv: "86873a46c3c60507e0da43aea4ac8581"
server_key_version: "encryptionkey@0"
I20260928 02:14:03.180991 20540 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790561432251112-7085-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790561432251112-7085-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790561432251112-7085-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:14:03.195397 20540 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:14:03.200412 20540 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:14:03.200678 20540 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:14:03.220131 20540 rpc_server.cc:307] RPC server started. Bound to: 127.6.235.126:41361
I20260928 02:14:03.220156 20606 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.235.126:41361 every 8 connection(s)
I20260928 02:14:03.221624 20540 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790561432251112-7085-0/minicluster-data/master-0/data/info.pb
I20260928 02:14:03.223970 7085 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTlaUuc/build/asan/bin/kudu as pid 20540
I20260928 02:14:03.224217 7085 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790561432251112-7085-0/minicluster-data/master-0/wal/instance
I20260928 02:14:03.225109 7085 external_mini_cluster.cc:1428] Setting key 906ecaed5f614af9a1c72c8b1076c351
I20260928 02:14:03.226476 20607 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:14:03.236109 20607 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.6.235.126" port: 41361 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.6.235.125@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.235.125@KRBTEST.COM" created.
Sep 28 02:14:03 dist-test-slave-86gv krb5kdc[20527](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561643, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.235.126@KRBTEST.COM for kudu/127.6.235.126@KRBTEST.COM
I20260928 02:14:03.263521 20611 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:14:03.240696 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37106 (local address 127.6.235.126:41361)
0928 02:14:03.241454 (+ 758us) server_negotiation.cc:206] Beginning negotiation
0928 02:14:03.241460 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 02:14:03.241509 (+ 49us) server_negotiation.cc:407] Connection header received
0928 02:14:03.243959 (+ 2450us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:14:03.243965 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:14:03.244215 (+ 250us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:14:03.244387 (+ 172us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:14:03.246205 (+ 1818us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:14:03.247488 (+ 1283us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:14:03.249771 (+ 2283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:14:03.250477 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:14:03.250626 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:14:03.254228 (+ 3602us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:14:03.254240 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:14:03.254262 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:14:03.254310 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:14:03.258405 (+ 4095us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:14:03.259295 (+ 890us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:14:03.259307 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:14:03.259313 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:14:03.259530 (+ 217us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:14:03.260059 (+ 529us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:14:03.260067 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:14:03.260073 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:14:03.260682 (+ 609us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:14:03.260949 (+ 267us) server_negotiation.cc:1035] Waiting for connection context
0928 02:14:03.261509 (+ 560us) server_negotiation.cc:299] Negotiation successful
0928 02:14:03.261952 (+ 443us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":24,"server-negotiator.queue_time_us":587,"thread_start_us":433,"threads_started":1}
I20260928 02:14:03.263530 20610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:14:03.239506 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.235.126:41361 (local address 127.0.0.1:37106)
0928 02:14:03.240422 (+ 916us) negotiation.cc:107] Waiting for socket to connect
0928 02:14:03.240478 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0928 02:14:03.243508 (+ 3030us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:14:03.244480 (+ 972us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:14:03.244490 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:14:03.245412 (+ 922us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:14:03.245968 (+ 556us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:14:03.246013 (+ 45us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:14:03.247671 (+ 1658us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:14:03.247680 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:14:03.249619 (+ 1939us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:14:03.249636 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:14:03.250695 (+ 1059us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:14:03.250700 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:14:03.250840 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:14:03.251465 (+ 625us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:14:03.251493 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:14:03.253953 (+ 2460us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:14:03.258569 (+ 4616us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:14:03.258580 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:14:03.258591 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:14:03.259133 (+ 542us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:14:03.259657 (+ 524us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:14:03.259662 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:14:03.259664 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:14:03.259926 (+ 262us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:14:03.260832 (+ 906us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:14:03.260840 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:14:03.261105 (+ 265us) client_negotiation.cc:769] Sending connection context
0928 02:14:03.261361 (+ 256us) client_negotiation.cc:240] Negotiation successful
0928 02:14:03.262076 (+ 715us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":604,"mutex_wait_us":223,"thread_start_us":363,"threads_started":1}
I20260928 02:14:03.267269 20607 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.6.235.125" port: 36985 } has no permanent_uuid. Determining permanent_uuid...
I20260928 02:14:03.268676 20610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:14:03.268219 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.235.125:36985 (local address 127.0.0.1:54570)
0928 02:14:03.268429 (+ 210us) negotiation.cc:107] Waiting for socket to connect
0928 02:14:03.268548 (+ 119us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.235.125:36985: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":78}
W20260928 02:14:03.269089 20558 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.235.125:36985: connect: Connection refused (error 111)
I20260928 02:14:03.270929 20610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:14:03.270475 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.235.125:36985 (local address 127.0.0.1:54578)
0928 02:14:03.270637 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0928 02:14:03.270737 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.235.125:36985: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260928 02:14:03.271155 20607 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.6.235.125:36985: Network error: Client connection negotiation failed: client connection to 127.6.235.125:36985: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.235.125 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790561432251112-7085-0/krb5kdc/kudu_127.6.235.125.keytab.
Entry for principal kudu/127.6.235.125 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790561432251112-7085-0/krb5kdc/kudu_127.6.235.125.keytab.
I20260928 02:14:03.284163 7085 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTlaUuc/build/asan/bin/kudu
/tmp/dist-test-taskTlaUuc/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790561432251112-7085-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790561432251112-7085-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790561432251112-7085-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790561432251112-7085-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.235.125:36985
--webserver_interface=127.6.235.125
--webserver_port=0
--keytab_file=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790561432251112-7085-0/krb5kdc/kudu_127.6.235.125.keytab
--principal=kudu/127.6.235.125
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.235.84:33969
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.235.126:41361,127.6.235.125:36985,127.6.235.124:45773
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790561432251112-7085-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790561432251112-7085-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790561432251112-7085-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 02:14:03.320463 20607 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.235.125" port: 36985 } attempt: 1
I20260928 02:14:03.321825 20610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:14:03.321287 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.235.125:36985 (local address 127.0.0.1:54594)
0928 02:14:03.321572 (+ 285us) negotiation.cc:107] Waiting for socket to connect
0928 02:14:03.321698 (+ 126us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.235.125:36985: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":131}
I20260928 02:14:03.322867 20610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:14:03.322487 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.235.125:36985 (local address 127.0.0.1:54606)
0928 02:14:03.322638 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0928 02:14:03.322716 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.235.125:36985: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260928 02:14:03.323097 20607 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.6.235.125:36985: Network error: Client connection negotiation failed: client connection to 127.6.235.125:36985: connect: Connection refused (error 111)
I20260928 02:14:03.391379 20607 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.235.125" port: 36985 } attempt: 2
I20260928 02:14:03.392592 20610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:14:03.392136 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.235.125:36985 (local address 127.0.0.1:54614)
0928 02:14:03.392343 (+ 207us) negotiation.cc:107] Waiting for socket to connect
0928 02:14:03.392440 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.235.125:36985: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":81}
I20260928 02:14:03.393625 20610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:14:03.393252 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.235.125:36985 (local address 127.0.0.1:54622)
0928 02:14:03.393418 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0928 02:14:03.393484 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.235.125:36985: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260928 02:14:03.393811 20607 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.6.235.125:36985: Network error: Client connection negotiation failed: client connection to 127.6.235.125:36985: connect: Connection refused (error 111)
I20260928 02:14:03.485090 20607 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.235.125" port: 36985 } attempt: 3
I20260928 02:14:03.486241 20610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:14:03.485789 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.235.125:36985 (local address 127.0.0.1:54624)
0928 02:14:03.485997 (+ 208us) negotiation.cc:107] Waiting for socket to connect
0928 02:14:03.486111 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.235.125:36985: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
I20260928 02:14:03.487282 20610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:14:03.486918 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.235.125:36985 (local address 127.0.0.1:54636)
0928 02:14:03.487082 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0928 02:14:03.487149 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.235.125:36985: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260928 02:14:03.487509 20607 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.6.235.125:36985: Network error: Client connection negotiation failed: client connection to 127.6.235.125:36985: connect: Connection refused (error 111)
I20260928 02:14:03.630767 20607 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.235.125" port: 36985 } attempt: 4
I20260928 02:14:03.632054 20610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:14:03.631432 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.235.125:36985 (local address 127.0.0.1:54638)
0928 02:14:03.631770 (+ 338us) negotiation.cc:107] Waiting for socket to connect
0928 02:14:03.631882 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.235.125:36985: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":171}
I20260928 02:14:03.633210 20610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:14:03.632753 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.235.125:36985 (local address 127.0.0.1:54646)
0928 02:14:03.632937 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0928 02:14:03.633051 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.235.125:36985: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260928 02:14:03.633445 20607 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.6.235.125:36985: Network error: Client connection negotiation failed: client connection to 127.6.235.125:36985: connect: Connection refused (error 111)
W20260928 02:14:03.638334 20614 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:14:03.638645 20614 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:14:03.638818 20614 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:14:03.646538 20614 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 02:14:03.646658 20614 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:14:03.646782 20614 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:14:03.646881 20614 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 02:14:03.646974 20614 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 02:14:03.657176 20614 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.235.84:33969
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790561432251112-7085-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790561432251112-7085-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.6.235.126:41361,127.6.235.125:36985,127.6.235.124:45773
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790561432251112-7085-0/krb5kdc/kudu_127.6.235.125.keytab
--principal=kudu/127.6.235.125
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.235.125:36985
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790561432251112-7085-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.6.235.125
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790561432251112-7085-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 00:51:59 UTC on ccycloud.quasar-axgkwn.root.comops.site
build id 4676464
ASAN enabled
I20260928 02:14:03.658601 20614 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:14:03.660148 20614 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:14:03.673014 20620 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:14:03.673043 20619 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:14:03.673105 20622 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:14:03.673801 20614 server_base.cc:1034] running on GCE node
I20260928 02:14:03.674609 20614 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:14:03.676098 20614 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:14:03.677309 20614 hybrid_clock.cc:648] HybridClock initialized: now 1790561643677265 us; error 40 us; skew 500 ppm
I20260928 02:14:03.677695 20614 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 02:14:03 dist-test-slave-86gv krb5kdc[20527](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561643, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.235.125@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:14:03.681052 20614 init.cc:377] Logged in from keytab as kudu/127.6.235.125@KRBTEST.COM (short username kudu)
I20260928 02:14:03.682608 20614 webserver.cc:460] Webserver started at http://127.6.235.125:37337/ using document root <none> and password file <none>
I20260928 02:14:03.683385 20614 fs_manager.cc:359] Metadata directory not provided
I20260928 02:14:03.683537 20614 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:14:03.683862 20614 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:14:03.687330 20614 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790561432251112-7085-0/minicluster-data/master-1/data/instance:
uuid: "f35bf367f7e84108bead421f89b0f86d"
format_stamp: "Formatted at 2026-09-28 02:14:03 on dist-test-slave-86gv"
server_key: "65f70b55add252be066722a0be1150cb"
server_key_iv: "e34ac4dd142637a2a6bc3b31423841ba"
server_key_version: "encryptionkey@0"
I20260928 02:14:03.688174 20614 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790561432251112-7085-0/minicluster-data/master-1/wal/instance:
uuid: "f35bf367f7e84108bead421f89b0f86d"
format_stamp: "Formatted at 2026-09-28 02:14:03 on dist-test-slave-86gv"
server_key: "65f70b55add252be066722a0be1150cb"
server_key_iv: "e34ac4dd142637a2a6bc3b31423841ba"
server_key_version: "encryptionkey@0"
I20260928 02:14:03.693629 20614 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260928 02:14:03.697250 20629 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:14:03.698741 20614 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.004s
I20260928 02:14:03.698940 20614 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790561432251112-7085-0/minicluster-data/master-1/data,/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790561432251112-7085-0/minicluster-data/master-1/wal
uuid: "f35bf367f7e84108bead421f89b0f86d"
format_stamp: "Formatted at 2026-09-28 02:14:03 on dist-test-slave-86gv"
server_key: "65f70b55add252be066722a0be1150cb"
server_key_iv: "e34ac4dd142637a2a6bc3b31423841ba"
server_key_version: "encryptionkey@0"
I20260928 02:14:03.699165 20614 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790561432251112-7085-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790561432251112-7085-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790561432251112-7085-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:14:03.724663 20614 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:14:03.729172 20614 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:14:03.729497 20614 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:14:03.749190 20614 rpc_server.cc:307] RPC server started. Bound to: 127.6.235.125:36985
I20260928 02:14:03.749214 20680 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.235.125:36985 every 8 connection(s)
I20260928 02:14:03.750849 20614 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790561432251112-7085-0/minicluster-data/master-1/data/info.pb
I20260928 02:14:03.754799 20681 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:14:03.758562 7085 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTlaUuc/build/asan/bin/kudu as pid 20614
I20260928 02:14:03.758786 7085 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790561432251112-7085-0/minicluster-data/master-1/wal/instance
I20260928 02:14:03.759229 7085 external_mini_cluster.cc:1428] Setting key 4fdd217f87f878942c4d088a943b7ae1
I20260928 02:14:03.761780 20681 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.6.235.126" port: 41361 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 02:14:03 dist-test-slave-86gv krb5kdc[20527](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561643, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.235.125@KRBTEST.COM for kudu/127.6.235.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.6.235.124@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.235.124@KRBTEST.COM" created.
I20260928 02:14:03.784297 20683 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:14:03.765190 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37114 (local address 127.6.235.126:41361)
0928 02:14:03.765825 (+ 635us) server_negotiation.cc:206] Beginning negotiation
0928 02:14:03.765830 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 02:14:03.766754 (+ 924us) server_negotiation.cc:407] Connection header received
0928 02:14:03.768719 (+ 1965us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:14:03.768728 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:14:03.768825 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:14:03.768989 (+ 164us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:14:03.770523 (+ 1534us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:14:03.771759 (+ 1236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:14:03.773418 (+ 1659us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:14:03.774131 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:14:03.774275 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:14:03.777811 (+ 3536us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:14:03.777816 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:14:03.777820 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:14:03.777875 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:14:03.780735 (+ 2860us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:14:03.781672 (+ 937us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:14:03.781676 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:14:03.781679 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:14:03.781820 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:14:03.782286 (+ 466us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:14:03.782289 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:14:03.782291 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:14:03.782596 (+ 305us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:14:03.782711 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0928 02:14:03.783669 (+ 958us) server_negotiation.cc:299] Negotiation successful
0928 02:14:03.783903 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":519,"thread_start_us":425,"threads_started":1}
I20260928 02:14:03.785496 20685 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:14:03.765633 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.235.126:41361 (local address 127.0.0.1:37114)
0928 02:14:03.766569 (+ 936us) negotiation.cc:107] Waiting for socket to connect
0928 02:14:03.766622 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0928 02:14:03.768419 (+ 1797us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:14:03.769187 (+ 768us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:14:03.769195 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:14:03.769776 (+ 581us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:14:03.770338 (+ 562us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:14:03.770354 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:14:03.771922 (+ 1568us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:14:03.771925 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:14:03.773265 (+ 1340us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:14:03.773277 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:14:03.774350 (+ 1073us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:14:03.774354 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:14:03.774549 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:14:03.775265 (+ 716us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:14:03.775287 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:14:03.777575 (+ 2288us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:14:03.780940 (+ 3365us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:14:03.780951 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:14:03.780979 (+ 28us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:14:03.781513 (+ 534us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:14:03.781940 (+ 427us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:14:03.781944 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:14:03.781946 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:14:03.782174 (+ 228us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:14:03.782717 (+ 543us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:14:03.782723 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:14:03.783202 (+ 479us) client_negotiation.cc:769] Sending connection context
0928 02:14:03.783536 (+ 334us) client_negotiation.cc:240] Negotiation successful
0928 02:14:03.784115 (+ 579us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":635,"thread_start_us":424,"threads_started":1}
I20260928 02:14:03.787292 20681 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.6.235.125" port: 36985 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 02:14:03 dist-test-slave-86gv krb5kdc[20527](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561643, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.235.125@KRBTEST.COM for kudu/127.6.235.125@KRBTEST.COM
I20260928 02:14:03.804294 20685 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:14:03.788136 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.235.125:36985 (local address 127.0.0.1:54656)
0928 02:14:03.788333 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0928 02:14:03.788354 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0928 02:14:03.788465 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:14:03.789578 (+ 1113us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:14:03.789586 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:14:03.790138 (+ 552us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:14:03.790348 (+ 210us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:14:03.790363 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:14:03.791820 (+ 1457us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:14:03.791827 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:14:03.792758 (+ 931us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:14:03.792769 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:14:03.794082 (+ 1313us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:14:03.794086 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:14:03.794260 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:14:03.794909 (+ 649us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:14:03.794926 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:14:03.797003 (+ 2077us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:14:03.801625 (+ 4622us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:14:03.801629 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:14:03.801631 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:14:03.801993 (+ 362us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:14:03.802440 (+ 447us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:14:03.802444 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:14:03.802446 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:14:03.802612 (+ 166us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:14:03.803444 (+ 832us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:14:03.803451 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:14:03.803597 (+ 146us) client_negotiation.cc:769] Sending connection context
0928 02:14:03.803712 (+ 115us) client_negotiation.cc:240] Negotiation successful
0928 02:14:03.803919 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":57}
I20260928 02:14:03.804739 20686 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:14:03.788377 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54656 (local address 127.6.235.125:36985)
0928 02:14:03.789142 (+ 765us) server_negotiation.cc:206] Beginning negotiation
0928 02:14:03.789148 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 02:14:03.789173 (+ 25us) server_negotiation.cc:407] Connection header received
0928 02:14:03.789214 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:14:03.789220 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:14:03.789397 (+ 177us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:14:03.789547 (+ 150us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:14:03.790519 (+ 972us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:14:03.791670 (+ 1151us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:14:03.792959 (+ 1289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:14:03.793939 (+ 980us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:14:03.794085 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:14:03.797231 (+ 3146us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:14:03.797248 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:14:03.797269 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:14:03.797312 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:14:03.801450 (+ 4138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:14:03.802139 (+ 689us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:14:03.802149 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:14:03.802154 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:14:03.802314 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:14:03.802732 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:14:03.802738 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:14:03.802742 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:14:03.803298 (+ 556us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:14:03.803571 (+ 273us) server_negotiation.cc:1035] Waiting for connection context
0928 02:14:03.803928 (+ 357us) server_negotiation.cc:299] Negotiation successful
0928 02:14:03.804242 (+ 314us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":607,"thread_start_us":429,"threads_started":1}
I20260928 02:14:03.807215 20681 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.6.235.124" port: 45773 } has no permanent_uuid. Determining permanent_uuid...
I20260928 02:14:03.808439 20685 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:14:03.807968 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.235.124:45773 (local address 127.0.0.1:38880)
0928 02:14:03.808176 (+ 208us) negotiation.cc:107] Waiting for socket to connect
0928 02:14:03.808301 (+ 125us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.235.124:45773: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":77}
W20260928 02:14:03.808770 20631 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.235.124:45773: connect: Connection refused (error 111)
I20260928 02:14:03.810328 20685 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:14:03.809958 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.235.124:45773 (local address 127.0.0.1:38890)
0928 02:14:03.810090 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0928 02:14:03.810192 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.235.124:45773: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260928 02:14:03.810542 20681 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.6.235.124:45773: Network error: Client connection negotiation failed: client connection to 127.6.235.124:45773: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.235.124 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790561432251112-7085-0/krb5kdc/kudu_127.6.235.124.keytab.
Entry for principal kudu/127.6.235.124 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790561432251112-7085-0/krb5kdc/kudu_127.6.235.124.keytab.
I20260928 02:14:03.817354 7085 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTlaUuc/build/asan/bin/kudu
/tmp/dist-test-taskTlaUuc/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790561432251112-7085-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790561432251112-7085-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790561432251112-7085-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790561432251112-7085-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.235.124:45773
--webserver_interface=127.6.235.124
--webserver_port=0
--keytab_file=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790561432251112-7085-0/krb5kdc/kudu_127.6.235.124.keytab
--principal=kudu/127.6.235.124
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.235.84:33969
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.235.126:41361,127.6.235.125:36985,127.6.235.124:45773
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790561432251112-7085-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790561432251112-7085-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790561432251112-7085-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 02:14:03.859886 20681 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.235.124" port: 45773 } attempt: 1
I20260928 02:14:03.861302 20685 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:14:03.860733 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.235.124:45773 (local address 127.0.0.1:38900)
0928 02:14:03.861032 (+ 299us) negotiation.cc:107] Waiting for socket to connect
0928 02:14:03.861146 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.235.124:45773: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":83}
I20260928 02:14:03.862370 20685 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:14:03.862028 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.235.124:45773 (local address 127.0.0.1:38902)
0928 02:14:03.862195 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0928 02:14:03.862261 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.235.124:45773: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260928 02:14:03.862591 20681 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.6.235.124:45773: Network error: Client connection negotiation failed: client connection to 127.6.235.124:45773: connect: Connection refused (error 111)
I20260928 02:14:03.930835 20681 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.235.124" port: 45773 } attempt: 2
I20260928 02:14:03.932130 20685 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:14:03.931663 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.235.124:45773 (local address 127.0.0.1:38918)
0928 02:14:03.931869 (+ 206us) negotiation.cc:107] Waiting for socket to connect
0928 02:14:03.931962 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.235.124:45773: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":82}
I20260928 02:14:03.932693 20607 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.235.125" port: 36985 } attempt: 5
I20260928 02:14:03.933218 20685 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:14:03.932782 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.235.124:45773 (local address 127.0.0.1:38922)
0928 02:14:03.932967 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0928 02:14:03.933051 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.235.124:45773: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260928 02:14:03.933676 20681 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.6.235.124:45773: Network error: Client connection negotiation failed: client connection to 127.6.235.124:45773: connect: Connection refused (error 111)
Sep 28 02:14:03 dist-test-slave-86gv krb5kdc[20527](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561643, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.235.126@KRBTEST.COM for kudu/127.6.235.125@KRBTEST.COM
I20260928 02:14:03.947769 20610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:14:03.933375 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.235.125:36985 (local address 127.0.0.1:54666)
0928 02:14:03.933559 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0928 02:14:03.933581 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0928 02:14:03.933720 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:14:03.934129 (+ 409us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:14:03.934134 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:14:03.934635 (+ 501us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:14:03.934907 (+ 272us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:14:03.934921 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:14:03.935897 (+ 976us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:14:03.935903 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:14:03.937364 (+ 1461us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:14:03.937384 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:14:03.938215 (+ 831us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:14:03.938221 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:14:03.938450 (+ 229us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:14:03.939147 (+ 697us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:14:03.939175 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:14:03.941702 (+ 2527us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:14:03.945427 (+ 3725us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:14:03.945432 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:14:03.945435 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:14:03.945839 (+ 404us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:14:03.946252 (+ 413us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:14:03.946255 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:14:03.946257 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:14:03.946392 (+ 135us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:14:03.947012 (+ 620us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:14:03.947016 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:14:03.947157 (+ 141us) client_negotiation.cc:769] Sending connection context
0928 02:14:03.947299 (+ 142us) client_negotiation.cc:240] Negotiation successful
0928 02:14:03.947521 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":41}
I20260928 02:14:03.947980 20686 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:14:03.933436 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54666 (local address 127.6.235.125:36985)
0928 02:14:03.933600 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0928 02:14:03.933606 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 02:14:03.933630 (+ 24us) server_negotiation.cc:407] Connection header received
0928 02:14:03.933897 (+ 267us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:14:03.933901 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:14:03.933975 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:14:03.934050 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:14:03.935067 (+ 1017us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:14:03.935753 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:14:03.937554 (+ 1801us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:14:03.938015 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:14:03.938116 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:14:03.941932 (+ 3816us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:14:03.941939 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:14:03.941944 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:14:03.941988 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:14:03.945254 (+ 3266us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:14:03.945970 (+ 716us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:14:03.945976 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:14:03.945980 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:14:03.946136 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:14:03.946507 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:14:03.946511 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:14:03.946515 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:14:03.946885 (+ 370us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:14:03.947035 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0928 02:14:03.947432 (+ 397us) server_negotiation.cc:299] Negotiation successful
0928 02:14:03.947682 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260928 02:14:03.949043 20607 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.6.235.124" port: 45773 } has no permanent_uuid. Determining permanent_uuid...
I20260928 02:14:03.950485 20610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:14:03.949916 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.235.124:45773 (local address 127.0.0.1:38930)
0928 02:14:03.950158 (+ 242us) negotiation.cc:107] Waiting for socket to connect
0928 02:14:03.950276 (+ 118us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.235.124:45773: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":76}
I20260928 02:14:03.952040 20610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:14:03.951591 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.235.124:45773 (local address 127.0.0.1:38942)
0928 02:14:03.951783 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0928 02:14:03.951903 (+ 120us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.235.124:45773: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
W20260928 02:14:03.952288 20607 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.6.235.124:45773: Network error: Client connection negotiation failed: client connection to 127.6.235.124:45773: connect: Connection refused (error 111)
I20260928 02:14:04.003567 20607 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.235.124" port: 45773 } attempt: 1
I20260928 02:14:04.005026 20610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:14:04.004400 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.235.124:45773 (local address 127.0.0.1:38956)
0928 02:14:04.004632 (+ 232us) negotiation.cc:107] Waiting for socket to connect
0928 02:14:04.004744 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.235.124:45773: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":85}
I20260928 02:14:04.006371 20610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:14:04.005864 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.235.124:45773 (local address 127.0.0.1:38964)
0928 02:14:04.006058 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0928 02:14:04.006189 (+ 131us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.235.124:45773: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
W20260928 02:14:04.006647 20607 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.6.235.124:45773: Network error: Client connection negotiation failed: client connection to 127.6.235.124:45773: connect: Connection refused (error 111)
I20260928 02:14:04.024986 20681 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.235.124" port: 45773 } attempt: 3
I20260928 02:14:04.026397 20685 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:14:04.025672 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.235.124:45773 (local address 127.0.0.1:38968)
0928 02:14:04.025961 (+ 289us) negotiation.cc:107] Waiting for socket to connect
0928 02:14:04.026101 (+ 140us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.235.124:45773: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":87}
I20260928 02:14:04.027705 20685 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:14:04.027125 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.235.124:45773 (local address 127.0.0.1:38980)
0928 02:14:04.027420 (+ 295us) negotiation.cc:107] Waiting for socket to connect
0928 02:14:04.027535 (+ 115us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.235.124:45773: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":150}
W20260928 02:14:04.027984 20681 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.6.235.124:45773: Network error: Client connection negotiation failed: client connection to 127.6.235.124:45773: connect: Connection refused (error 111)
I20260928 02:14:04.074894 20607 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.235.124" port: 45773 } attempt: 2
I20260928 02:14:04.076272 20610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:14:04.075606 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.235.124:45773 (local address 127.0.0.1:38982)
0928 02:14:04.075902 (+ 296us) negotiation.cc:107] Waiting for socket to connect
0928 02:14:04.076039 (+ 137us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.235.124:45773: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":91}
I20260928 02:14:04.077629 20610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:14:04.077168 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.235.124:45773 (local address 127.0.0.1:38994)
0928 02:14:04.077364 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0928 02:14:04.077481 (+ 117us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.235.124:45773: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
W20260928 02:14:04.077896 20607 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.6.235.124:45773: Network error: Client connection negotiation failed: client connection to 127.6.235.124:45773: connect: Connection refused (error 111)
I20260928 02:14:04.171258 20681 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.235.124" port: 45773 } attempt: 4
I20260928 02:14:04.172500 20685 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:14:04.171947 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.235.124:45773 (local address 127.0.0.1:39004)
0928 02:14:04.172195 (+ 248us) negotiation.cc:107] Waiting for socket to connect
0928 02:14:04.172299 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.235.124:45773: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":73}
I20260928 02:14:04.173610 20685 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:14:04.173254 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.235.124:45773 (local address 127.0.0.1:39006)
0928 02:14:04.173415 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0928 02:14:04.173492 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.235.124:45773: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260928 02:14:04.173827 20681 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.6.235.124:45773: Network error: Client connection negotiation failed: client connection to 127.6.235.124:45773: connect: Connection refused (error 111)
W20260928 02:14:04.174244 20689 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:14:04.174571 20689 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:14:04.174724 20689 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:14:04.182443 20689 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 02:14:04.182608 20689 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:14:04.182715 20689 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:14:04.182819 20689 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 02:14:04.182907 20689 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 02:14:04.184124 20607 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.235.124" port: 45773 } attempt: 3
I20260928 02:14:04.185421 20610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:14:04.184868 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.235.124:45773 (local address 127.0.0.1:39018)
0928 02:14:04.185107 (+ 239us) negotiation.cc:107] Waiting for socket to connect
0928 02:14:04.185215 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.235.124:45773: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":90}
I20260928 02:14:04.186640 20610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:14:04.186186 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.235.124:45773 (local address 127.0.0.1:39022)
0928 02:14:04.186420 (+ 234us) negotiation.cc:107] Waiting for socket to connect
0928 02:14:04.186505 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.235.124:45773: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":97}
W20260928 02:14:04.186844 20607 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.6.235.124:45773: Network error: Client connection negotiation failed: client connection to 127.6.235.124:45773: connect: Connection refused (error 111)
I20260928 02:14:04.193964 20689 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.235.84:33969
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790561432251112-7085-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790561432251112-7085-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.6.235.126:41361,127.6.235.125:36985,127.6.235.124:45773
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790561432251112-7085-0/krb5kdc/kudu_127.6.235.124.keytab
--principal=kudu/127.6.235.124
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.235.124:45773
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790561432251112-7085-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.6.235.124
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790561432251112-7085-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 00:51:59 UTC on ccycloud.quasar-axgkwn.root.comops.site
build id 4676464
ASAN enabled
I20260928 02:14:04.195346 20689 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:14:04.196828 20689 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:14:04.209002 20697 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:14:04.209066 20695 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:14:04.209431 20694 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:14:04.214567 20689 server_base.cc:1034] running on GCE node
I20260928 02:14:04.215405 20689 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:14:04.216884 20689 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:14:04.218174 20689 hybrid_clock.cc:648] HybridClock initialized: now 1790561644218130 us; error 30 us; skew 500 ppm
I20260928 02:14:04.218494 20689 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 02:14:04 dist-test-slave-86gv krb5kdc[20527](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561644, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.235.124@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:14:04.222152 20689 init.cc:377] Logged in from keytab as kudu/127.6.235.124@KRBTEST.COM (short username kudu)
I20260928 02:14:04.223757 20689 webserver.cc:460] Webserver started at http://127.6.235.124:43755/ using document root <none> and password file <none>
I20260928 02:14:04.224469 20689 fs_manager.cc:359] Metadata directory not provided
I20260928 02:14:04.224606 20689 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:14:04.224954 20689 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:14:04.227995 20689 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790561432251112-7085-0/minicluster-data/master-2/data/instance:
uuid: "60acd7aa8f004cc3875f5fa8a3cb3acd"
format_stamp: "Formatted at 2026-09-28 02:14:04 on dist-test-slave-86gv"
server_key: "e7d809b2112161fb34d9881712606e9f"
server_key_iv: "19e2bc6298ba19b7af5ff421a3e9c1f3"
server_key_version: "encryptionkey@0"
I20260928 02:14:04.228822 20689 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790561432251112-7085-0/minicluster-data/master-2/wal/instance:
uuid: "60acd7aa8f004cc3875f5fa8a3cb3acd"
format_stamp: "Formatted at 2026-09-28 02:14:04 on dist-test-slave-86gv"
server_key: "e7d809b2112161fb34d9881712606e9f"
server_key_iv: "19e2bc6298ba19b7af5ff421a3e9c1f3"
server_key_version: "encryptionkey@0"
I20260928 02:14:04.234664 20689 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260928 02:14:04.238358 20704 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:14:04.239753 20689 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260928 02:14:04.239938 20689 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790561432251112-7085-0/minicluster-data/master-2/data,/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790561432251112-7085-0/minicluster-data/master-2/wal
uuid: "60acd7aa8f004cc3875f5fa8a3cb3acd"
format_stamp: "Formatted at 2026-09-28 02:14:04 on dist-test-slave-86gv"
server_key: "e7d809b2112161fb34d9881712606e9f"
server_key_iv: "19e2bc6298ba19b7af5ff421a3e9c1f3"
server_key_version: "encryptionkey@0"
I20260928 02:14:04.240180 20689 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790561432251112-7085-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790561432251112-7085-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790561432251112-7085-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:14:04.257946 20689 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:14:04.262427 20689 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:14:04.262748 20689 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:14:04.282227 20689 rpc_server.cc:307] RPC server started. Bound to: 127.6.235.124:45773
I20260928 02:14:04.282259 20755 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.235.124:45773 every 8 connection(s)
I20260928 02:14:04.283739 20689 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790561432251112-7085-0/minicluster-data/master-2/data/info.pb
I20260928 02:14:04.287807 20756 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:14:04.291643 7085 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTlaUuc/build/asan/bin/kudu as pid 20689
I20260928 02:14:04.291831 7085 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790561432251112-7085-0/minicluster-data/master-2/wal/instance
I20260928 02:14:04.292403 7085 external_mini_cluster.cc:1428] Setting key cdf223983b0b4bd11ef3a23d384a44b5
I20260928 02:14:04.295630 20756 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.6.235.126" port: 41361 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 02:14:04 dist-test-slave-86gv krb5kdc[20527](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561644, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.235.124@KRBTEST.COM for kudu/127.6.235.126@KRBTEST.COM
Sep 28 02:14:04 dist-test-slave-86gv krb5kdc[20527](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561642, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.235.124@KRBTEST.COM
I20260928 02:14:04.317342 20760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:14:04.299415 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37130 (local address 127.6.235.126:41361)
0928 02:14:04.299917 (+ 502us) server_negotiation.cc:206] Beginning negotiation
0928 02:14:04.299924 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 02:14:04.300526 (+ 602us) server_negotiation.cc:407] Connection header received
0928 02:14:04.300704 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:14:04.300709 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:14:04.300795 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:14:04.301049 (+ 254us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:14:04.302528 (+ 1479us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:14:04.303688 (+ 1160us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:14:04.305632 (+ 1944us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:14:04.306272 (+ 640us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:14:04.306381 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:14:04.309919 (+ 3538us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:14:04.309924 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:14:04.309927 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:14:04.309971 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:14:04.313457 (+ 3486us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:14:04.314624 (+ 1167us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:14:04.314633 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:14:04.314637 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:14:04.314809 (+ 172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:14:04.315343 (+ 534us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:14:04.315351 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:14:04.315354 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:14:04.315714 (+ 360us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:14:04.315868 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0928 02:14:04.316670 (+ 802us) server_negotiation.cc:299] Negotiation successful
0928 02:14:04.316900 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":344,"thread_start_us":251,"threads_started":1}
I20260928 02:14:04.318320 20761 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:14:04.299506 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.235.126:41361 (local address 127.0.0.1:37130)
0928 02:14:04.300349 (+ 843us) negotiation.cc:107] Waiting for socket to connect
0928 02:14:04.300399 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0928 02:14:04.300571 (+ 172us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:14:04.301160 (+ 589us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:14:04.301167 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:14:04.301955 (+ 788us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:14:04.302344 (+ 389us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:14:04.302354 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:14:04.303898 (+ 1544us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:14:04.303903 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:14:04.305463 (+ 1560us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:14:04.305476 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:14:04.306519 (+ 1043us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:14:04.306530 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:14:04.306671 (+ 141us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:14:04.307265 (+ 594us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:14:04.307287 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:14:04.309698 (+ 2411us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:14:04.313621 (+ 3923us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:14:04.313631 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:14:04.313641 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:14:04.314457 (+ 816us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:14:04.314945 (+ 488us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:14:04.314950 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:14:04.314953 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:14:04.315224 (+ 271us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:14:04.315846 (+ 622us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:14:04.315852 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:14:04.316249 (+ 397us) client_negotiation.cc:769] Sending connection context
0928 02:14:04.316580 (+ 331us) client_negotiation.cc:240] Negotiation successful
0928 02:14:04.317156 (+ 576us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":585,"mutex_wait_us":1,"spinlock_wait_cycles":12032,"thread_start_us":413,"threads_started":1}
I20260928 02:14:04.318339 20759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:14:04.294380 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39034 (local address 127.6.235.124:45773)
0928 02:14:04.295191 (+ 811us) server_negotiation.cc:206] Beginning negotiation
0928 02:14:04.295203 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0928 02:14:04.295244 (+ 41us) server_negotiation.cc:407] Connection header received
0928 02:14:04.296824 (+ 1580us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:14:04.296847 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:14:04.297401 (+ 554us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:14:04.297923 (+ 522us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:14:04.300166 (+ 2243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:14:04.302283 (+ 2117us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:14:04.304323 (+ 2040us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:14:04.305440 (+ 1117us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:14:04.305650 (+ 210us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:14:04.310438 (+ 4788us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:14:04.310452 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:14:04.310472 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:14:04.310540 (+ 68us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:14:04.314171 (+ 3631us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:14:04.315110 (+ 939us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:14:04.315116 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:14:04.315119 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:14:04.315261 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:14:04.315666 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:14:04.315673 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:14:04.315676 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:14:04.316258 (+ 582us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:14:04.316468 (+ 210us) server_negotiation.cc:1035] Waiting for connection context
0928 02:14:04.316990 (+ 522us) server_negotiation.cc:299] Negotiation successful
0928 02:14:04.317271 (+ 281us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":101,"server-negotiator.queue_time_us":609,"spinlock_wait_cycles":16640,"thread_start_us":387,"threads_started":1}
I20260928 02:14:04.320185 20756 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.6.235.125" port: 36985 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 02:14:04 dist-test-slave-86gv krb5kdc[20527](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561644, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.235.124@KRBTEST.COM for kudu/127.6.235.125@KRBTEST.COM
Sep 28 02:14:04 dist-test-slave-86gv krb5kdc[20527](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561642, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.235.126@KRBTEST.COM
I20260928 02:14:04.335219 20761 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:14:04.321020 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.235.125:36985 (local address 127.0.0.1:54678)
0928 02:14:04.321203 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0928 02:14:04.321224 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0928 02:14:04.321354 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:14:04.321717 (+ 363us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:14:04.321721 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:14:04.322153 (+ 432us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:14:04.322363 (+ 210us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:14:04.322376 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:14:04.323794 (+ 1418us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:14:04.323797 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:14:04.325072 (+ 1275us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:14:04.325088 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:14:04.325970 (+ 882us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:14:04.325974 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:14:04.326134 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:14:04.326742 (+ 608us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:14:04.326758 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:14:04.329151 (+ 2393us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:14:04.332699 (+ 3548us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:14:04.332705 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:14:04.332709 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:14:04.333257 (+ 548us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:14:04.333719 (+ 462us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:14:04.333723 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:14:04.333725 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:14:04.333870 (+ 145us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:14:04.334532 (+ 662us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:14:04.334535 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:14:04.334669 (+ 134us) client_negotiation.cc:769] Sending connection context
0928 02:14:04.334774 (+ 105us) client_negotiation.cc:240] Negotiation successful
0928 02:14:04.334981 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":62}
I20260928 02:14:04.335556 20686 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:14:04.321155 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54678 (local address 127.6.235.125:36985)
0928 02:14:04.321363 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0928 02:14:04.321369 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 02:14:04.321388 (+ 19us) server_negotiation.cc:407] Connection header received
0928 02:14:04.321452 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:14:04.321458 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:14:04.321550 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:14:04.321639 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:14:04.322529 (+ 890us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:14:04.323643 (+ 1114us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:14:04.325214 (+ 1571us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:14:04.325823 (+ 609us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:14:04.325958 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:14:04.329364 (+ 3406us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:14:04.329373 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:14:04.329380 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:14:04.329431 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:14:04.332558 (+ 3127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:14:04.333412 (+ 854us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:14:04.333421 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:14:04.333426 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:14:04.333598 (+ 172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:14:04.333991 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:14:04.333996 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:14:04.333999 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:14:04.334384 (+ 385us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:14:04.334565 (+ 181us) server_negotiation.cc:1035] Waiting for connection context
0928 02:14:04.334942 (+ 377us) server_negotiation.cc:299] Negotiation successful
0928 02:14:04.335158 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260928 02:14:04.336556 20756 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.6.235.124" port: 45773 } has no permanent_uuid. Determining permanent_uuid...
I20260928 02:14:04.338814 20760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:14:04.322433 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37138 (local address 127.6.235.126:41361)
0928 02:14:04.322596 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0928 02:14:04.322600 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:14:04.322612 (+ 12us) server_negotiation.cc:407] Connection header received
0928 02:14:04.322652 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:14:04.322656 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:14:04.322714 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:14:04.322786 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:14:04.324270 (+ 1484us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:14:04.325283 (+ 1013us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:14:04.326926 (+ 1643us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:14:04.327583 (+ 657us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:14:04.327712 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:14:04.332550 (+ 4838us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:14:04.332555 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:14:04.332558 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:14:04.332591 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:14:04.335869 (+ 3278us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:14:04.336730 (+ 861us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:14:04.336736 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:14:04.336739 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:14:04.336885 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:14:04.337391 (+ 506us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:14:04.337398 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:14:04.337403 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:14:04.337817 (+ 414us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:14:04.338089 (+ 272us) server_negotiation.cc:1035] Waiting for connection context
0928 02:14:04.338401 (+ 312us) server_negotiation.cc:299] Negotiation successful
0928 02:14:04.338577 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
Sep 28 02:14:04 dist-test-slave-86gv krb5kdc[20527](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561644, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.235.124@KRBTEST.COM for kudu/127.6.235.124@KRBTEST.COM
Sep 28 02:14:04 dist-test-slave-86gv krb5kdc[20527](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561642, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.235.125@KRBTEST.COM
I20260928 02:14:04.351405 20761 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:14:04.337567 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.235.124:45773 (local address 127.0.0.1:39044)
0928 02:14:04.337703 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0928 02:14:04.337717 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 02:14:04.337834 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:14:04.338192 (+ 358us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:14:04.338199 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:14:04.338680 (+ 481us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:14:04.338856 (+ 176us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:14:04.338869 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:14:04.340112 (+ 1243us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:14:04.340116 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:14:04.341381 (+ 1265us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:14:04.341393 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:14:04.342411 (+ 1018us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:14:04.342415 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:14:04.342620 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:14:04.343241 (+ 621us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:14:04.343262 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:14:04.345628 (+ 2366us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:14:04.348646 (+ 3018us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:14:04.348654 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:14:04.348657 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:14:04.349231 (+ 574us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:14:04.349675 (+ 444us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:14:04.349681 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:14:04.349685 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:14:04.349855 (+ 170us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:14:04.350371 (+ 516us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:14:04.350375 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:14:04.350572 (+ 197us) client_negotiation.cc:769] Sending connection context
0928 02:14:04.350686 (+ 114us) client_negotiation.cc:240] Negotiation successful
0928 02:14:04.351015 (+ 329us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":54}
I20260928 02:14:04.351673 20759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:14:04.337403 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39044 (local address 127.6.235.124:45773)
0928 02:14:04.337582 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0928 02:14:04.337586 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:14:04.337781 (+ 195us) server_negotiation.cc:407] Connection header received
0928 02:14:04.337961 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:14:04.337967 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:14:04.338050 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:14:04.338142 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:14:04.339028 (+ 886us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:14:04.339966 (+ 938us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:14:04.341542 (+ 1576us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:14:04.342256 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:14:04.342386 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:14:04.345808 (+ 3422us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:14:04.345814 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:14:04.345819 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:14:04.345856 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:14:04.348513 (+ 2657us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:14:04.349354 (+ 841us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:14:04.349358 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:14:04.349360 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:14:04.349556 (+ 196us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:14:04.349971 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:14:04.349974 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:14:04.349976 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:14:04.350248 (+ 272us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:14:04.350355 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0928 02:14:04.350835 (+ 480us) server_negotiation.cc:299] Negotiation successful
0928 02:14:04.351218 (+ 383us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":111,"server-negotiator.queue_time_us":53}
I20260928 02:14:04.357008 20686 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:14:04.340636 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54692 (local address 127.6.235.125:36985)
0928 02:14:04.340802 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0928 02:14:04.340809 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 02:14:04.340827 (+ 18us) server_negotiation.cc:407] Connection header received
0928 02:14:04.340929 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:14:04.340935 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:14:04.341007 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:14:04.341101 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:14:04.342569 (+ 1468us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:14:04.343713 (+ 1144us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:14:04.345473 (+ 1760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:14:04.345921 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:14:04.346065 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:14:04.350567 (+ 4502us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:14:04.350572 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:14:04.350576 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:14:04.350607 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:14:04.354290 (+ 3683us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:14:04.354937 (+ 647us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:14:04.354944 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:14:04.354948 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:14:04.355127 (+ 179us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:14:04.355507 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:14:04.355513 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:14:04.355516 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:14:04.355840 (+ 324us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:14:04.356141 (+ 301us) server_negotiation.cc:1035] Waiting for connection context
0928 02:14:04.356419 (+ 278us) server_negotiation.cc:299] Negotiation successful
0928 02:14:04.356643 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260928 02:14:04.357769 20756 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 60acd7aa8f004cc3875f5fa8a3cb3acd: Bootstrap starting.
I20260928 02:14:04.361601 20756 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 60acd7aa8f004cc3875f5fa8a3cb3acd: Neither blocks nor log segments found. Creating new log.
I20260928 02:14:04.362253 7085 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 02:14:04.362869 20756 log.cc:824] T 00000000000000000000000000000000 P 60acd7aa8f004cc3875f5fa8a3cb3acd: Log is configured to *not* fsync() on all Append() calls
I20260928 02:14:04.364087 20607 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.235.124" port: 45773 } attempt: 4
I20260928 02:14:04.366397 20756 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 60acd7aa8f004cc3875f5fa8a3cb3acd: No bootstrap required, opened a new log
Sep 28 02:14:04 dist-test-slave-86gv krb5kdc[20527](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561643, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.235.126@KRBTEST.COM for kudu/127.6.235.124@KRBTEST.COM
I20260928 02:14:04.394177 20756 raft_consensus.cc:348] T 00000000000000000000000000000000 P 60acd7aa8f004cc3875f5fa8a3cb3acd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "680b03dcbe2a4f85b1d399dd62507518" member_type: VOTER last_known_addr { host: "127.6.235.126" port: 41361 } } peers { permanent_uuid: "f35bf367f7e84108bead421f89b0f86d" member_type: VOTER last_known_addr { host: "127.6.235.125" port: 36985 } } peers { permanent_uuid: "60acd7aa8f004cc3875f5fa8a3cb3acd" member_type: VOTER last_known_addr { host: "127.6.235.124" port: 45773 } }
I20260928 02:14:04.394683 20756 raft_consensus.cc:374] T 00000000000000000000000000000000 P 60acd7aa8f004cc3875f5fa8a3cb3acd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:14:04.394814 20756 raft_consensus.cc:729] T 00000000000000000000000000000000 P 60acd7aa8f004cc3875f5fa8a3cb3acd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 60acd7aa8f004cc3875f5fa8a3cb3acd, State: Initialized, Role: FOLLOWER
I20260928 02:14:04.396502 20610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:14:04.364739 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.235.124:45773 (local address 127.0.0.1:39056)
0928 02:14:04.364953 (+ 214us) negotiation.cc:107] Waiting for socket to connect
0928 02:14:04.364973 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0928 02:14:04.365102 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:14:04.365454 (+ 352us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:14:04.365462 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:14:04.365875 (+ 413us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:14:04.366106 (+ 231us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:14:04.366123 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:14:04.367676 (+ 1553us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:14:04.367681 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:14:04.368981 (+ 1300us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:14:04.368997 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:14:04.370640 (+ 1643us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:14:04.370651 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:14:04.370972 (+ 321us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:14:04.371906 (+ 934us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:14:04.371941 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:14:04.375619 (+ 3678us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:14:04.387701 (+ 12082us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:14:04.387711 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:14:04.387718 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:14:04.388342 (+ 624us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:14:04.388853 (+ 511us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:14:04.388859 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:14:04.388862 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:14:04.389071 (+ 209us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:14:04.395357 (+ 6286us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:14:04.395365 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:14:04.395590 (+ 225us) client_negotiation.cc:769] Sending connection context
0928 02:14:04.395735 (+ 145us) client_negotiation.cc:240] Negotiation successful
0928 02:14:04.396059 (+ 324us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":50}
I20260928 02:14:04.397823 20759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:14:04.364780 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39056 (local address 127.6.235.124:45773)
0928 02:14:04.365021 (+ 241us) server_negotiation.cc:206] Beginning negotiation
0928 02:14:04.365025 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:14:04.365042 (+ 17us) server_negotiation.cc:407] Connection header received
0928 02:14:04.365244 (+ 202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:14:04.365248 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:14:04.365306 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:14:04.365412 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:14:04.366305 (+ 893us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:14:04.367522 (+ 1217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:14:04.369368 (+ 1846us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:14:04.370396 (+ 1028us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:14:04.371407 (+ 1011us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:14:04.381095 (+ 9688us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:14:04.381112 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:14:04.381120 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:14:04.381186 (+ 66us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:14:04.386730 (+ 5544us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:14:04.388528 (+ 1798us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:14:04.388533 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:14:04.388537 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:14:04.388724 (+ 187us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:14:04.389212 (+ 488us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:14:04.389216 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:14:04.389220 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:14:04.389682 (+ 462us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:14:04.396979 (+ 7297us) server_negotiation.cc:1035] Waiting for connection context
0928 02:14:04.397131 (+ 152us) server_negotiation.cc:299] Negotiation successful
0928 02:14:04.397420 (+ 289us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260928 02:14:04.404448 20607 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 680b03dcbe2a4f85b1d399dd62507518: Bootstrap starting.
I20260928 02:14:04.404675 20760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:14:04.369063 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37144 (local address 127.6.235.126:41361)
0928 02:14:04.369251 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0928 02:14:04.369257 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 02:14:04.384510 (+ 15253us) server_negotiation.cc:407] Connection header received
0928 02:14:04.384767 (+ 257us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:14:04.384777 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:14:04.384869 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:14:04.385025 (+ 156us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:14:04.387803 (+ 2778us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:14:04.389034 (+ 1231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:14:04.391451 (+ 2417us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:14:04.392187 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:14:04.392290 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:14:04.396396 (+ 4106us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:14:04.396402 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:14:04.396407 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:14:04.396451 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:14:04.400089 (+ 3638us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:14:04.402108 (+ 2019us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:14:04.402114 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:14:04.402118 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:14:04.402263 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:14:04.402702 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:14:04.402708 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:14:04.402711 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:14:04.403087 (+ 376us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:14:04.403387 (+ 300us) server_negotiation.cc:1035] Waiting for connection context
0928 02:14:04.404022 (+ 635us) server_negotiation.cc:299] Negotiation successful
0928 02:14:04.404272 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260928 02:14:04.408452 20686 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:14:04.369232 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54702 (local address 127.6.235.125:36985)
0928 02:14:04.374821 (+ 5589us) server_negotiation.cc:206] Beginning negotiation
0928 02:14:04.374828 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 02:14:04.387474 (+ 12646us) server_negotiation.cc:407] Connection header received
0928 02:14:04.387639 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:14:04.387648 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:14:04.387763 (+ 115us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:14:04.387905 (+ 142us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:14:04.390735 (+ 2830us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:14:04.391965 (+ 1230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:14:04.394314 (+ 2349us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:14:04.395022 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:14:04.395128 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:14:04.399712 (+ 4584us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:14:04.399723 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:14:04.399729 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:14:04.399787 (+ 58us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:14:04.403434 (+ 3647us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:14:04.404348 (+ 914us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:14:04.404354 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:14:04.404357 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:14:04.404543 (+ 186us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:14:04.405587 (+ 1044us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:14:04.405592 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:14:04.405596 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:14:04.406964 (+ 1368us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:14:04.407250 (+ 286us) server_negotiation.cc:1035] Waiting for connection context
0928 02:14:04.407837 (+ 587us) server_negotiation.cc:299] Negotiation successful
0928 02:14:04.408096 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5431}
I20260928 02:14:04.408681 20607 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 680b03dcbe2a4f85b1d399dd62507518: Neither blocks nor log segments found. Creating new log.
I20260928 02:14:04.409525 20756 consensus_queue.cc:260] T 00000000000000000000000000000000 P 60acd7aa8f004cc3875f5fa8a3cb3acd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "680b03dcbe2a4f85b1d399dd62507518" member_type: VOTER last_known_addr { host: "127.6.235.126" port: 41361 } } peers { permanent_uuid: "f35bf367f7e84108bead421f89b0f86d" member_type: VOTER last_known_addr { host: "127.6.235.125" port: 36985 } } peers { permanent_uuid: "60acd7aa8f004cc3875f5fa8a3cb3acd" member_type: VOTER last_known_addr { host: "127.6.235.124" port: 45773 } }
I20260928 02:14:04.410156 20769 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:14:04.373183 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39072 (local address 127.6.235.124:45773)
0928 02:14:04.381370 (+ 8187us) server_negotiation.cc:206] Beginning negotiation
0928 02:14:04.381380 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 02:14:04.389245 (+ 7865us) server_negotiation.cc:407] Connection header received
0928 02:14:04.389773 (+ 528us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:14:04.389779 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:14:04.389866 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:14:04.389961 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:14:04.392386 (+ 2425us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:14:04.393719 (+ 1333us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:14:04.396204 (+ 2485us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:14:04.396958 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:14:04.397073 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:14:04.400510 (+ 3437us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:14:04.400521 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:14:04.400530 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:14:04.400580 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:14:04.405629 (+ 5049us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:14:04.406540 (+ 911us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:14:04.406546 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:14:04.406550 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:14:04.406706 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:14:04.407242 (+ 536us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:14:04.407247 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:14:04.407250 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:14:04.407610 (+ 360us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:14:04.407830 (+ 220us) server_negotiation.cc:1035] Waiting for connection context
0928 02:14:04.408418 (+ 588us) server_negotiation.cc:299] Negotiation successful
0928 02:14:04.409766 (+ 1348us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7984,"thread_start_us":7711,"threads_started":1}
I20260928 02:14:04.410130 20607 log.cc:824] T 00000000000000000000000000000000 P 680b03dcbe2a4f85b1d399dd62507518: Log is configured to *not* fsync() on all Append() calls
I20260928 02:14:04.411254 20771 sys_catalog.cc:455] T 00000000000000000000000000000000 P 60acd7aa8f004cc3875f5fa8a3cb3acd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "680b03dcbe2a4f85b1d399dd62507518" member_type: VOTER last_known_addr { host: "127.6.235.126" port: 41361 } } peers { permanent_uuid: "f35bf367f7e84108bead421f89b0f86d" member_type: VOTER last_known_addr { host: "127.6.235.125" port: 36985 } } peers { permanent_uuid: "60acd7aa8f004cc3875f5fa8a3cb3acd" member_type: VOTER last_known_addr { host: "127.6.235.124" port: 45773 } } }
I20260928 02:14:04.412604 20771 sys_catalog.cc:458] T 00000000000000000000000000000000 P 60acd7aa8f004cc3875f5fa8a3cb3acd [sys.catalog]: This master's current role is: FOLLOWER
I20260928 02:14:04.412727 20756 sys_catalog.cc:564] T 00000000000000000000000000000000 P 60acd7aa8f004cc3875f5fa8a3cb3acd [sys.catalog]: configured and running, proceeding with master startup.
I20260928 02:14:04.414441 20607 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 680b03dcbe2a4f85b1d399dd62507518: No bootstrap required, opened a new log
I20260928 02:14:04.425184 20607 raft_consensus.cc:348] T 00000000000000000000000000000000 P 680b03dcbe2a4f85b1d399dd62507518 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "680b03dcbe2a4f85b1d399dd62507518" member_type: VOTER last_known_addr { host: "127.6.235.126" port: 41361 } } peers { permanent_uuid: "f35bf367f7e84108bead421f89b0f86d" member_type: VOTER last_known_addr { host: "127.6.235.125" port: 36985 } } peers { permanent_uuid: "60acd7aa8f004cc3875f5fa8a3cb3acd" member_type: VOTER last_known_addr { host: "127.6.235.124" port: 45773 } }
I20260928 02:14:04.425680 20607 raft_consensus.cc:374] T 00000000000000000000000000000000 P 680b03dcbe2a4f85b1d399dd62507518 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:14:04.425815 20607 raft_consensus.cc:729] T 00000000000000000000000000000000 P 680b03dcbe2a4f85b1d399dd62507518 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 680b03dcbe2a4f85b1d399dd62507518, State: Initialized, Role: FOLLOWER
I20260928 02:14:04.426625 20607 consensus_queue.cc:260] T 00000000000000000000000000000000 P 680b03dcbe2a4f85b1d399dd62507518 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "680b03dcbe2a4f85b1d399dd62507518" member_type: VOTER last_known_addr { host: "127.6.235.126" port: 41361 } } peers { permanent_uuid: "f35bf367f7e84108bead421f89b0f86d" member_type: VOTER last_known_addr { host: "127.6.235.125" port: 36985 } } peers { permanent_uuid: "60acd7aa8f004cc3875f5fa8a3cb3acd" member_type: VOTER last_known_addr { host: "127.6.235.124" port: 45773 } }
W20260928 02:14:04.429206 20783 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 60acd7aa8f004cc3875f5fa8a3cb3acd: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 02:14:04.429412 20783 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 02:14:04.430111 20785 sys_catalog.cc:455] T 00000000000000000000000000000000 P 680b03dcbe2a4f85b1d399dd62507518 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "680b03dcbe2a4f85b1d399dd62507518" member_type: VOTER last_known_addr { host: "127.6.235.126" port: 41361 } } peers { permanent_uuid: "f35bf367f7e84108bead421f89b0f86d" member_type: VOTER last_known_addr { host: "127.6.235.125" port: 36985 } } peers { permanent_uuid: "60acd7aa8f004cc3875f5fa8a3cb3acd" member_type: VOTER last_known_addr { host: "127.6.235.124" port: 45773 } } }
I20260928 02:14:04.430632 20785 sys_catalog.cc:458] T 00000000000000000000000000000000 P 680b03dcbe2a4f85b1d399dd62507518 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 02:14:04.431543 20607 sys_catalog.cc:564] T 00000000000000000000000000000000 P 680b03dcbe2a4f85b1d399dd62507518 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 02:14:04.421614 20689 master_runner.cc:186] Error getting master registration for 127.6.235.126:41361: OK, instance_id { permanent_uuid: "680b03dcbe2a4f85b1d399dd62507518" instance_seqno: 1790561643214689 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260928 02:14:04.435189 20689 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260928 02:14:04.445621 20796 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 680b03dcbe2a4f85b1d399dd62507518: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 02:14:04.445775 20796 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
W20260928 02:14:04.453660 20561 tablet.cc:2369] T 00000000000000000000000000000000 P 680b03dcbe2a4f85b1d399dd62507518: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 02:14:04.445883 20540 master_runner.cc:186] Error getting master registration for 127.6.235.125:36985: OK, instance_id { permanent_uuid: "f35bf367f7e84108bead421f89b0f86d" instance_seqno: 1790561643743601 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260928 02:14:04.458961 20540 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260928 02:14:04.473061 20681 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.235.124" port: 45773 } attempt: 5
Sep 28 02:14:04 dist-test-slave-86gv krb5kdc[20527](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561643, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.235.125@KRBTEST.COM for kudu/127.6.235.124@KRBTEST.COM
I20260928 02:14:04.491324 20685 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:14:04.473783 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.235.124:45773 (local address 127.0.0.1:39078)
0928 02:14:04.474119 (+ 336us) negotiation.cc:107] Waiting for socket to connect
0928 02:14:04.474141 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0928 02:14:04.474278 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:14:04.474766 (+ 488us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:14:04.474771 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:14:04.475386 (+ 615us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:14:04.475630 (+ 244us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:14:04.475643 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:14:04.477187 (+ 1544us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:14:04.477193 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:14:04.478681 (+ 1488us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:14:04.478699 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:14:04.479738 (+ 1039us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:14:04.479745 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:14:04.479966 (+ 221us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:14:04.480707 (+ 741us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:14:04.480733 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:14:04.483735 (+ 3002us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:14:04.488303 (+ 4568us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:14:04.488310 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:14:04.488315 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:14:04.488936 (+ 621us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:14:04.489404 (+ 468us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:14:04.489409 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:14:04.489413 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:14:04.489598 (+ 185us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:14:04.490230 (+ 632us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:14:04.490235 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:14:04.490411 (+ 176us) client_negotiation.cc:769] Sending connection context
0928 02:14:04.490590 (+ 179us) client_negotiation.cc:240] Negotiation successful
0928 02:14:04.490918 (+ 328us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":165}
I20260928 02:14:04.491346 20769 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:14:04.473891 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39078 (local address 127.6.235.124:45773)
0928 02:14:04.474122 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0928 02:14:04.474129 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 02:14:04.474216 (+ 87us) server_negotiation.cc:407] Connection header received
0928 02:14:04.474444 (+ 228us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:14:04.474451 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:14:04.474578 (+ 127us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:14:04.474694 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:14:04.475827 (+ 1133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:14:04.477015 (+ 1188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:14:04.478870 (+ 1855us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:14:04.479564 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:14:04.479700 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:14:04.483978 (+ 4278us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:14:04.483986 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:14:04.483993 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:14:04.484043 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:14:04.488120 (+ 4077us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:14:04.489074 (+ 954us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:14:04.489081 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:14:04.489085 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:14:04.489265 (+ 180us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:14:04.489723 (+ 458us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:14:04.489728 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:14:04.489732 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:14:04.490094 (+ 362us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:14:04.490250 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0928 02:14:04.490734 (+ 484us) server_negotiation.cc:299] Negotiation successful
0928 02:14:04.490981 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260928 02:14:04.499266 20681 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f35bf367f7e84108bead421f89b0f86d: Bootstrap starting.
I20260928 02:14:04.503785 20681 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f35bf367f7e84108bead421f89b0f86d: Neither blocks nor log segments found. Creating new log.
I20260928 02:14:04.505275 20681 log.cc:824] T 00000000000000000000000000000000 P f35bf367f7e84108bead421f89b0f86d: Log is configured to *not* fsync() on all Append() calls
I20260928 02:14:04.510450 20681 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f35bf367f7e84108bead421f89b0f86d: No bootstrap required, opened a new log
W20260928 02:14:04.514993 20710 tablet.cc:2369] T 00000000000000000000000000000000 P 60acd7aa8f004cc3875f5fa8a3cb3acd: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 02:14:04.522099 20681 raft_consensus.cc:348] T 00000000000000000000000000000000 P f35bf367f7e84108bead421f89b0f86d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "680b03dcbe2a4f85b1d399dd62507518" member_type: VOTER last_known_addr { host: "127.6.235.126" port: 41361 } } peers { permanent_uuid: "f35bf367f7e84108bead421f89b0f86d" member_type: VOTER last_known_addr { host: "127.6.235.125" port: 36985 } } peers { permanent_uuid: "60acd7aa8f004cc3875f5fa8a3cb3acd" member_type: VOTER last_known_addr { host: "127.6.235.124" port: 45773 } }
I20260928 02:14:04.522578 20681 raft_consensus.cc:374] T 00000000000000000000000000000000 P f35bf367f7e84108bead421f89b0f86d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:14:04.522691 20681 raft_consensus.cc:729] T 00000000000000000000000000000000 P f35bf367f7e84108bead421f89b0f86d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f35bf367f7e84108bead421f89b0f86d, State: Initialized, Role: FOLLOWER
I20260928 02:14:04.523507 20681 consensus_queue.cc:260] T 00000000000000000000000000000000 P f35bf367f7e84108bead421f89b0f86d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "680b03dcbe2a4f85b1d399dd62507518" member_type: VOTER last_known_addr { host: "127.6.235.126" port: 41361 } } peers { permanent_uuid: "f35bf367f7e84108bead421f89b0f86d" member_type: VOTER last_known_addr { host: "127.6.235.125" port: 36985 } } peers { permanent_uuid: "60acd7aa8f004cc3875f5fa8a3cb3acd" member_type: VOTER last_known_addr { host: "127.6.235.124" port: 45773 } }
I20260928 02:14:04.526732 20799 sys_catalog.cc:455] T 00000000000000000000000000000000 P f35bf367f7e84108bead421f89b0f86d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "680b03dcbe2a4f85b1d399dd62507518" member_type: VOTER last_known_addr { host: "127.6.235.126" port: 41361 } } peers { permanent_uuid: "f35bf367f7e84108bead421f89b0f86d" member_type: VOTER last_known_addr { host: "127.6.235.125" port: 36985 } } peers { permanent_uuid: "60acd7aa8f004cc3875f5fa8a3cb3acd" member_type: VOTER last_known_addr { host: "127.6.235.124" port: 45773 } } }
I20260928 02:14:04.527249 20799 sys_catalog.cc:458] T 00000000000000000000000000000000 P f35bf367f7e84108bead421f89b0f86d [sys.catalog]: This master's current role is: FOLLOWER
I20260928 02:14:04.528193 20681 sys_catalog.cc:564] T 00000000000000000000000000000000 P f35bf367f7e84108bead421f89b0f86d [sys.catalog]: configured and running, proceeding with master startup.
I20260928 02:14:04.545109 20614 master_runner.cc:263] No leader master found from master f35bf367f7e84108bead421f89b0f86d
I20260928 02:14:04.545279 20614 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260928 02:14:04.548312 20810 catalog_manager.cc:1558] T 00000000000000000000000000000000 P f35bf367f7e84108bead421f89b0f86d: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 02:14:04.548590 20810 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 02:14:04.612290 20785 raft_consensus.cc:482] T 00000000000000000000000000000000 P 680b03dcbe2a4f85b1d399dd62507518 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 02:14:04.612703 20785 raft_consensus.cc:504] T 00000000000000000000000000000000 P 680b03dcbe2a4f85b1d399dd62507518 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "680b03dcbe2a4f85b1d399dd62507518" member_type: VOTER last_known_addr { host: "127.6.235.126" port: 41361 } } peers { permanent_uuid: "f35bf367f7e84108bead421f89b0f86d" member_type: VOTER last_known_addr { host: "127.6.235.125" port: 36985 } } peers { permanent_uuid: "60acd7aa8f004cc3875f5fa8a3cb3acd" member_type: VOTER last_known_addr { host: "127.6.235.124" port: 45773 } }
I20260928 02:14:04.614531 20785 leader_election.cc:288] T 00000000000000000000000000000000 P 680b03dcbe2a4f85b1d399dd62507518 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f35bf367f7e84108bead421f89b0f86d (127.6.235.125:36985), 60acd7aa8f004cc3875f5fa8a3cb3acd (127.6.235.124:45773)
I20260928 02:14:04.615628 20731 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "680b03dcbe2a4f85b1d399dd62507518" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "60acd7aa8f004cc3875f5fa8a3cb3acd" is_pre_election: true
I20260928 02:14:04.616281 20731 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 60acd7aa8f004cc3875f5fa8a3cb3acd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 680b03dcbe2a4f85b1d399dd62507518 in term 0.
I20260928 02:14:04.617115 20557 leader_election.cc:302] T 00000000000000000000000000000000 P 680b03dcbe2a4f85b1d399dd62507518 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60acd7aa8f004cc3875f5fa8a3cb3acd, 680b03dcbe2a4f85b1d399dd62507518; no voters:
I20260928 02:14:04.618348 20656 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "680b03dcbe2a4f85b1d399dd62507518" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f35bf367f7e84108bead421f89b0f86d" is_pre_election: true
I20260928 02:14:04.619004 20656 raft_consensus.cc:2454] T 00000000000000000000000000000000 P f35bf367f7e84108bead421f89b0f86d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 680b03dcbe2a4f85b1d399dd62507518 in term 0.
I20260928 02:14:04.623958 20785 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 680b03dcbe2a4f85b1d399dd62507518 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 02:14:04.624143 20785 raft_consensus.cc:482] T 00000000000000000000000000000000 P 680b03dcbe2a4f85b1d399dd62507518 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 02:14:04.624248 20785 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 680b03dcbe2a4f85b1d399dd62507518 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:14:04.628131 20785 raft_consensus.cc:504] T 00000000000000000000000000000000 P 680b03dcbe2a4f85b1d399dd62507518 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "680b03dcbe2a4f85b1d399dd62507518" member_type: VOTER last_known_addr { host: "127.6.235.126" port: 41361 } } peers { permanent_uuid: "f35bf367f7e84108bead421f89b0f86d" member_type: VOTER last_known_addr { host: "127.6.235.125" port: 36985 } } peers { permanent_uuid: "60acd7aa8f004cc3875f5fa8a3cb3acd" member_type: VOTER last_known_addr { host: "127.6.235.124" port: 45773 } }
I20260928 02:14:04.629561 20656 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "680b03dcbe2a4f85b1d399dd62507518" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f35bf367f7e84108bead421f89b0f86d"
I20260928 02:14:04.629808 20656 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f35bf367f7e84108bead421f89b0f86d [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:14:04.633615 20656 raft_consensus.cc:2454] T 00000000000000000000000000000000 P f35bf367f7e84108bead421f89b0f86d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 680b03dcbe2a4f85b1d399dd62507518 in term 1.
I20260928 02:14:04.634222 20558 leader_election.cc:302] T 00000000000000000000000000000000 P 680b03dcbe2a4f85b1d399dd62507518 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 680b03dcbe2a4f85b1d399dd62507518, f35bf367f7e84108bead421f89b0f86d; no voters:
I20260928 02:14:04.636758 20731 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "680b03dcbe2a4f85b1d399dd62507518" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "60acd7aa8f004cc3875f5fa8a3cb3acd"
I20260928 02:14:04.637040 20731 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 60acd7aa8f004cc3875f5fa8a3cb3acd [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:14:04.640533 20731 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 60acd7aa8f004cc3875f5fa8a3cb3acd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 680b03dcbe2a4f85b1d399dd62507518 in term 1.
I20260928 02:14:04.641460 20785 leader_election.cc:288] T 00000000000000000000000000000000 P 680b03dcbe2a4f85b1d399dd62507518 [CANDIDATE]: Term 1 election: Requested vote from peers f35bf367f7e84108bead421f89b0f86d (127.6.235.125:36985), 60acd7aa8f004cc3875f5fa8a3cb3acd (127.6.235.124:45773)
I20260928 02:14:04.641922 20785 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 680b03dcbe2a4f85b1d399dd62507518 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:14:04.642338 20785 raft_consensus.cc:686] T 00000000000000000000000000000000 P 680b03dcbe2a4f85b1d399dd62507518 [term 1 LEADER]: Becoming Leader. State: Replica: 680b03dcbe2a4f85b1d399dd62507518, State: Running, Role: LEADER
I20260928 02:14:04.643198 20785 consensus_queue.cc:237] T 00000000000000000000000000000000 P 680b03dcbe2a4f85b1d399dd62507518 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "680b03dcbe2a4f85b1d399dd62507518" member_type: VOTER last_known_addr { host: "127.6.235.126" port: 41361 } } peers { permanent_uuid: "f35bf367f7e84108bead421f89b0f86d" member_type: VOTER last_known_addr { host: "127.6.235.125" port: 36985 } } peers { permanent_uuid: "60acd7aa8f004cc3875f5fa8a3cb3acd" member_type: VOTER last_known_addr { host: "127.6.235.124" port: 45773 } }
I20260928 02:14:04.664383 20812 sys_catalog.cc:455] T 00000000000000000000000000000000 P 680b03dcbe2a4f85b1d399dd62507518 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 680b03dcbe2a4f85b1d399dd62507518. Latest consensus state: current_term: 1 leader_uuid: "680b03dcbe2a4f85b1d399dd62507518" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "680b03dcbe2a4f85b1d399dd62507518" member_type: VOTER last_known_addr { host: "127.6.235.126" port: 41361 } } peers { permanent_uuid: "f35bf367f7e84108bead421f89b0f86d" member_type: VOTER last_known_addr { host: "127.6.235.125" port: 36985 } } peers { permanent_uuid: "60acd7aa8f004cc3875f5fa8a3cb3acd" member_type: VOTER last_known_addr { host: "127.6.235.124" port: 45773 } } }
I20260928 02:14:04.665117 20812 sys_catalog.cc:458] T 00000000000000000000000000000000 P 680b03dcbe2a4f85b1d399dd62507518 [sys.catalog]: This master's current role is: LEADER
I20260928 02:14:04.672577 20814 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 02:14:04.675279 20814 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 02:14:04.696434 20731 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 60acd7aa8f004cc3875f5fa8a3cb3acd [term 1 FOLLOWER]: Refusing update from remote peer 680b03dcbe2a4f85b1d399dd62507518: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 02:14:04.697717 20812 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 680b03dcbe2a4f85b1d399dd62507518 [LEADER]: Connected to new peer: Peer: permanent_uuid: "60acd7aa8f004cc3875f5fa8a3cb3acd" member_type: VOTER last_known_addr { host: "127.6.235.124" port: 45773 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 02:14:04.705662 20656 raft_consensus.cc:1261] T 00000000000000000000000000000000 P f35bf367f7e84108bead421f89b0f86d [term 1 FOLLOWER]: Refusing update from remote peer 680b03dcbe2a4f85b1d399dd62507518: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 02:14:04.706810 20812 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 680b03dcbe2a4f85b1d399dd62507518 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f35bf367f7e84108bead421f89b0f86d" member_type: VOTER last_known_addr { host: "127.6.235.125" port: 36985 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260928 02:14:04.732183 20635 tablet.cc:2369] T 00000000000000000000000000000000 P f35bf367f7e84108bead421f89b0f86d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 02:14:04.738049 20799 sys_catalog.cc:455] T 00000000000000000000000000000000 P f35bf367f7e84108bead421f89b0f86d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 680b03dcbe2a4f85b1d399dd62507518. Latest consensus state: current_term: 1 leader_uuid: "680b03dcbe2a4f85b1d399dd62507518" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "680b03dcbe2a4f85b1d399dd62507518" member_type: VOTER last_known_addr { host: "127.6.235.126" port: 41361 } } peers { permanent_uuid: "f35bf367f7e84108bead421f89b0f86d" member_type: VOTER last_known_addr { host: "127.6.235.125" port: 36985 } } peers { permanent_uuid: "60acd7aa8f004cc3875f5fa8a3cb3acd" member_type: VOTER last_known_addr { host: "127.6.235.124" port: 45773 } } }
I20260928 02:14:04.738389 20799 sys_catalog.cc:458] T 00000000000000000000000000000000 P f35bf367f7e84108bead421f89b0f86d [sys.catalog]: This master's current role is: FOLLOWER
I20260928 02:14:04.741276 20815 mvcc.cc:205] Tried to move back new op lower bound from 7334140496663785472 to 7334140496496885760. Current Snapshot: MvccSnapshot[applied={T|T < 7334140496663785472}]
I20260928 02:14:04.744503 20785 sys_catalog.cc:455] T 00000000000000000000000000000000 P 680b03dcbe2a4f85b1d399dd62507518 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "680b03dcbe2a4f85b1d399dd62507518" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "680b03dcbe2a4f85b1d399dd62507518" member_type: VOTER last_known_addr { host: "127.6.235.126" port: 41361 } } peers { permanent_uuid: "f35bf367f7e84108bead421f89b0f86d" member_type: VOTER last_known_addr { host: "127.6.235.125" port: 36985 } } peers { permanent_uuid: "60acd7aa8f004cc3875f5fa8a3cb3acd" member_type: VOTER last_known_addr { host: "127.6.235.124" port: 45773 } } }
I20260928 02:14:04.744863 20785 sys_catalog.cc:458] T 00000000000000000000000000000000 P 680b03dcbe2a4f85b1d399dd62507518 [sys.catalog]: This master's current role is: LEADER
I20260928 02:14:04.746539 20817 mvcc.cc:205] Tried to move back new op lower bound from 7334140496663785472 to 7334140496496885760. Current Snapshot: MvccSnapshot[applied={T|T < 7334140496663785472}]
I20260928 02:14:04.747429 20814 catalog_manager.cc:1347] Generated new cluster ID: 8816246c00474933b5c1d6005210d097
I20260928 02:14:04.747556 20814 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 02:14:04.757316 20799 sys_catalog.cc:455] T 00000000000000000000000000000000 P f35bf367f7e84108bead421f89b0f86d [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "680b03dcbe2a4f85b1d399dd62507518" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "680b03dcbe2a4f85b1d399dd62507518" member_type: VOTER last_known_addr { host: "127.6.235.126" port: 41361 } } peers { permanent_uuid: "f35bf367f7e84108bead421f89b0f86d" member_type: VOTER last_known_addr { host: "127.6.235.125" port: 36985 } } peers { permanent_uuid: "60acd7aa8f004cc3875f5fa8a3cb3acd" member_type: VOTER last_known_addr { host: "127.6.235.124" port: 45773 } } }
I20260928 02:14:04.757624 20799 sys_catalog.cc:458] T 00000000000000000000000000000000 P f35bf367f7e84108bead421f89b0f86d [sys.catalog]: This master's current role is: FOLLOWER
I20260928 02:14:04.765344 20771 sys_catalog.cc:455] T 00000000000000000000000000000000 P 60acd7aa8f004cc3875f5fa8a3cb3acd [sys.catalog]: SysCatalogTable state changed. Reason: New leader 680b03dcbe2a4f85b1d399dd62507518. Latest consensus state: current_term: 1 leader_uuid: "680b03dcbe2a4f85b1d399dd62507518" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "680b03dcbe2a4f85b1d399dd62507518" member_type: VOTER last_known_addr { host: "127.6.235.126" port: 41361 } } peers { permanent_uuid: "f35bf367f7e84108bead421f89b0f86d" member_type: VOTER last_known_addr { host: "127.6.235.125" port: 36985 } } peers { permanent_uuid: "60acd7aa8f004cc3875f5fa8a3cb3acd" member_type: VOTER last_known_addr { host: "127.6.235.124" port: 45773 } } }
I20260928 02:14:04.765724 20771 sys_catalog.cc:458] T 00000000000000000000000000000000 P 60acd7aa8f004cc3875f5fa8a3cb3acd [sys.catalog]: This master's current role is: FOLLOWER
I20260928 02:14:04.781831 20812 sys_catalog.cc:455] T 00000000000000000000000000000000 P 680b03dcbe2a4f85b1d399dd62507518 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "680b03dcbe2a4f85b1d399dd62507518" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "680b03dcbe2a4f85b1d399dd62507518" member_type: VOTER last_known_addr { host: "127.6.235.126" port: 41361 } } peers { permanent_uuid: "f35bf367f7e84108bead421f89b0f86d" member_type: VOTER last_known_addr { host: "127.6.235.125" port: 36985 } } peers { permanent_uuid: "60acd7aa8f004cc3875f5fa8a3cb3acd" member_type: VOTER last_known_addr { host: "127.6.235.124" port: 45773 } } }
I20260928 02:14:04.782146 20812 sys_catalog.cc:458] T 00000000000000000000000000000000 P 680b03dcbe2a4f85b1d399dd62507518 [sys.catalog]: This master's current role is: LEADER
I20260928 02:14:04.784631 20816 mvcc.cc:205] Tried to move back new op lower bound from 7334140496663785472 to 7334140496496885760. Current Snapshot: MvccSnapshot[applied={T|T < 7334140496663785472}]
I20260928 02:14:04.800649 20771 sys_catalog.cc:455] T 00000000000000000000000000000000 P 60acd7aa8f004cc3875f5fa8a3cb3acd [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "680b03dcbe2a4f85b1d399dd62507518" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "680b03dcbe2a4f85b1d399dd62507518" member_type: VOTER last_known_addr { host: "127.6.235.126" port: 41361 } } peers { permanent_uuid: "f35bf367f7e84108bead421f89b0f86d" member_type: VOTER last_known_addr { host: "127.6.235.125" port: 36985 } } peers { permanent_uuid: "60acd7aa8f004cc3875f5fa8a3cb3acd" member_type: VOTER last_known_addr { host: "127.6.235.124" port: 45773 } } }
I20260928 02:14:04.800997 20771 sys_catalog.cc:458] T 00000000000000000000000000000000 P 60acd7aa8f004cc3875f5fa8a3cb3acd [sys.catalog]: This master's current role is: FOLLOWER
I20260928 02:14:04.803135 20814 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 02:14:04.805226 20814 catalog_manager.cc:1504] Loading token signing keys...
I20260928 02:14:04.836534 20814 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 680b03dcbe2a4f85b1d399dd62507518: Generated new TSK 0
I20260928 02:14:04.844755 20814 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 02:14:05.431403 20783 catalog_manager.cc:1259] Loaded cluster ID: 8816246c00474933b5c1d6005210d097
I20260928 02:14:05.431519 20783 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 60acd7aa8f004cc3875f5fa8a3cb3acd: loading cluster ID for follower catalog manager: success
I20260928 02:14:05.434186 20783 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 60acd7aa8f004cc3875f5fa8a3cb3acd: acquiring CA information for follower catalog manager: success
I20260928 02:14:05.436151 20783 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 60acd7aa8f004cc3875f5fa8a3cb3acd: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260928 02:14:05.550695 20810 catalog_manager.cc:1259] Loaded cluster ID: 8816246c00474933b5c1d6005210d097
I20260928 02:14:05.550803 20810 catalog_manager.cc:1552] T 00000000000000000000000000000000 P f35bf367f7e84108bead421f89b0f86d: loading cluster ID for follower catalog manager: success
I20260928 02:14:05.553328 20810 catalog_manager.cc:1574] T 00000000000000000000000000000000 P f35bf367f7e84108bead421f89b0f86d: acquiring CA information for follower catalog manager: success
I20260928 02:14:05.555187 20810 catalog_manager.cc:1602] T 00000000000000000000000000000000 P f35bf367f7e84108bead421f89b0f86d: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260928 02:14:06.003546 7085 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskTlaUuc/build/asan/bin/kudu pid 20540
I20260928 02:14:06.004863 20602 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 02:14:06.119115 7085 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTlaUuc/build/asan/bin/kudu with pid 20540
I20260928 02:14:06.141340 7085 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskTlaUuc/build/asan/bin/kudu pid 20614
I20260928 02:14:06.142460 20676 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 02:14:06.248869 7085 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTlaUuc/build/asan/bin/kudu with pid 20614
I20260928 02:14:06.270093 7085 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskTlaUuc/build/asan/bin/kudu pid 20689
I20260928 02:14:06.271359 20751 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 02:14:06.379442 7085 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTlaUuc/build/asan/bin/kudu with pid 20689
2026-09-28T02:14:06Z chronyd exiting
[ OK ] SecurityITest.IPKICACert (5030 ms)
[ RUN ] SecurityITest.IPKICACertWebServerEndPoints
Loading random data
Initializing database '/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790561432251112-7085-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 02:14:06 dist-test-slave-86gv krb5kdc[20835](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 02:14:06 dist-test-slave-86gv krb5kdc[20835](info): set up 2 sockets
Sep 28 02:14:06 dist-test-slave-86gv krb5kdc[20835](info): commencing operation
krb5kdc: starting...
W20260928 02:14:07.501693 7085 mini_kdc.cc:121] Time spent starting KDC: real 1.080s user 0.002s sys 0.026s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790561432251112-7085-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790561432251112-7085-0/krb5kdc/test-user.keytab.
Sep 28 02:14:07 dist-test-slave-86gv krb5kdc[20835](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561647, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T02:14:07Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T02:14:07Z Disabled control of system clock
WARNING: no policy specified for kudu/127.6.235.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.235.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.235.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790561432251112-7085-0/krb5kdc/kudu_127.6.235.126.keytab.
Entry for principal kudu/127.6.235.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790561432251112-7085-0/krb5kdc/kudu_127.6.235.126.keytab.
I20260928 02:14:07.752583 7085 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTlaUuc/build/asan/bin/kudu
/tmp/dist-test-taskTlaUuc/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790561432251112-7085-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790561432251112-7085-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790561432251112-7085-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790561432251112-7085-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.235.126:43607
--webserver_interface=127.6.235.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790561432251112-7085-0/krb5kdc/kudu_127.6.235.126.keytab
--principal=kudu/127.6.235.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.235.84:43753
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.235.126:43607
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790561432251112-7085-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790561432251112-7085-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790561432251112-7085-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:14:08.095067 20848 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:14:08.095407 20848 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:14:08.095587 20848 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:14:08.102871 20848 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 02:14:08.103055 20848 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:14:08.103169 20848 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:14:08.103255 20848 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 02:14:08.103343 20848 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 02:14:08.113467 20848 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.235.84:43753
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790561432251112-7085-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790561432251112-7085-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.235.126:43607
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790561432251112-7085-0/krb5kdc/kudu_127.6.235.126.keytab
--principal=kudu/127.6.235.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.235.126:43607
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790561432251112-7085-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.235.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790561432251112-7085-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 00:51:59 UTC on ccycloud.quasar-axgkwn.root.comops.site
build id 4676464
ASAN enabled
I20260928 02:14:08.115006 20848 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:14:08.116366 20848 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:14:08.129438 20853 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:14:08.129521 20856 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:14:08.129539 20854 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:14:08.130239 20848 server_base.cc:1034] running on GCE node
I20260928 02:14:08.131006 20848 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:14:08.132447 20848 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:14:08.133661 20848 hybrid_clock.cc:648] HybridClock initialized: now 1790561648133596 us; error 57 us; skew 500 ppm
I20260928 02:14:08.134058 20848 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 02:14:08 dist-test-slave-86gv krb5kdc[20835](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561648, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.235.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:14:08.137480 20848 init.cc:377] Logged in from keytab as kudu/127.6.235.126@KRBTEST.COM (short username kudu)
I20260928 02:14:08.139074 20848 webserver.cc:460] Webserver started at http://127.6.235.126:34123/ using document root <none> and password file <none>
I20260928 02:14:08.139801 20848 fs_manager.cc:359] Metadata directory not provided
I20260928 02:14:08.139940 20848 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:14:08.140290 20848 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:14:08.143489 20848 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790561432251112-7085-0/minicluster-data/master-0/data/instance:
uuid: "2fa77fc2d09d498993a3bf423b6eb5f6"
format_stamp: "Formatted at 2026-09-28 02:14:08 on dist-test-slave-86gv"
server_key: "2b017760ab5e7e8686a9652ef7d37ded"
server_key_iv: "a500373fe27d09dd6b6af62d38d02ea1"
server_key_version: "encryptionkey@0"
I20260928 02:14:08.144299 20848 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790561432251112-7085-0/minicluster-data/master-0/wal/instance:
uuid: "2fa77fc2d09d498993a3bf423b6eb5f6"
format_stamp: "Formatted at 2026-09-28 02:14:08 on dist-test-slave-86gv"
server_key: "2b017760ab5e7e8686a9652ef7d37ded"
server_key_iv: "a500373fe27d09dd6b6af62d38d02ea1"
server_key_version: "encryptionkey@0"
I20260928 02:14:08.149875 20848 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260928 02:14:08.153705 20863 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:14:08.155331 20848 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260928 02:14:08.155642 20848 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790561432251112-7085-0/minicluster-data/master-0/data,/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790561432251112-7085-0/minicluster-data/master-0/wal
uuid: "2fa77fc2d09d498993a3bf423b6eb5f6"
format_stamp: "Formatted at 2026-09-28 02:14:08 on dist-test-slave-86gv"
server_key: "2b017760ab5e7e8686a9652ef7d37ded"
server_key_iv: "a500373fe27d09dd6b6af62d38d02ea1"
server_key_version: "encryptionkey@0"
I20260928 02:14:08.155884 20848 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790561432251112-7085-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790561432251112-7085-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790561432251112-7085-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:14:08.170528 20848 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:14:08.175324 20848 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:14:08.175623 20848 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:14:08.195899 20848 rpc_server.cc:307] RPC server started. Bound to: 127.6.235.126:43607
I20260928 02:14:08.195956 20914 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.235.126:43607 every 8 connection(s)
I20260928 02:14:08.197506 20848 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790561432251112-7085-0/minicluster-data/master-0/data/info.pb
I20260928 02:14:08.201728 20915 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:14:08.206449 7085 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTlaUuc/build/asan/bin/kudu as pid 20848
I20260928 02:14:08.206704 7085 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790561432251112-7085-0/minicluster-data/master-0/wal/instance
I20260928 02:14:08.207340 7085 external_mini_cluster.cc:1428] Setting key 012b5d4a817454acac834f04ddf957c7
I20260928 02:14:08.212816 20915 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2fa77fc2d09d498993a3bf423b6eb5f6: Bootstrap starting.
I20260928 02:14:08.216670 20915 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2fa77fc2d09d498993a3bf423b6eb5f6: Neither blocks nor log segments found. Creating new log.
I20260928 02:14:08.218019 20915 log.cc:824] T 00000000000000000000000000000000 P 2fa77fc2d09d498993a3bf423b6eb5f6: Log is configured to *not* fsync() on all Append() calls
I20260928 02:14:08.221753 20915 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2fa77fc2d09d498993a3bf423b6eb5f6: No bootstrap required, opened a new log
Sep 28 02:14:08 dist-test-slave-86gv krb5kdc[20835](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561647, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.235.126@KRBTEST.COM
I20260928 02:14:08.232681 20915 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2fa77fc2d09d498993a3bf423b6eb5f6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2fa77fc2d09d498993a3bf423b6eb5f6" member_type: VOTER last_known_addr { host: "127.6.235.126" port: 43607 } }
I20260928 02:14:08.233215 20915 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2fa77fc2d09d498993a3bf423b6eb5f6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:14:08.233330 20915 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2fa77fc2d09d498993a3bf423b6eb5f6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2fa77fc2d09d498993a3bf423b6eb5f6, State: Initialized, Role: FOLLOWER
I20260928 02:14:08.233500 20918 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:14:08.209286 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47938 (local address 127.6.235.126:43607)
0928 02:14:08.210178 (+ 892us) server_negotiation.cc:206] Beginning negotiation
0928 02:14:08.210194 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0928 02:14:08.210241 (+ 47us) server_negotiation.cc:407] Connection header received
0928 02:14:08.211976 (+ 1735us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:14:08.211999 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:14:08.212405 (+ 406us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:14:08.212741 (+ 336us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:14:08.214656 (+ 1915us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:14:08.215966 (+ 1310us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:14:08.217954 (+ 1988us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:14:08.218919 (+ 965us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:14:08.219125 (+ 206us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:14:08.224001 (+ 4876us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:14:08.224013 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:14:08.224034 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:14:08.224084 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:14:08.228381 (+ 4297us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:14:08.229363 (+ 982us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:14:08.229374 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:14:08.229389 (+ 15us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:14:08.229614 (+ 225us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:14:08.229984 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:14:08.229991 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:14:08.230000 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:14:08.230844 (+ 844us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:14:08.231166 (+ 322us) server_negotiation.cc:1035] Waiting for connection context
0928 02:14:08.231549 (+ 383us) server_negotiation.cc:299] Negotiation successful
0928 02:14:08.232014 (+ 465us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":640,"thread_start_us":411,"threads_started":1}
I20260928 02:14:08.234122 20915 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2fa77fc2d09d498993a3bf423b6eb5f6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2fa77fc2d09d498993a3bf423b6eb5f6" member_type: VOTER last_known_addr { host: "127.6.235.126" port: 43607 } }
I20260928 02:14:08.234426 20915 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2fa77fc2d09d498993a3bf423b6eb5f6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 02:14:08.234565 20915 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2fa77fc2d09d498993a3bf423b6eb5f6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 02:14:08.234755 20915 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2fa77fc2d09d498993a3bf423b6eb5f6 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:14:08.237498 20915 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2fa77fc2d09d498993a3bf423b6eb5f6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2fa77fc2d09d498993a3bf423b6eb5f6" member_type: VOTER last_known_addr { host: "127.6.235.126" port: 43607 } }
I20260928 02:14:08.237888 7085 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 02:14:08.238094 20915 leader_election.cc:302] T 00000000000000000000000000000000 P 2fa77fc2d09d498993a3bf423b6eb5f6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2fa77fc2d09d498993a3bf423b6eb5f6; no voters:
I20260928 02:14:08.238744 20915 leader_election.cc:288] T 00000000000000000000000000000000 P 2fa77fc2d09d498993a3bf423b6eb5f6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 02:14:08.238931 20920 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2fa77fc2d09d498993a3bf423b6eb5f6 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:14:08.240782 20915 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2fa77fc2d09d498993a3bf423b6eb5f6 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 02:14:08.241575 20920 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2fa77fc2d09d498993a3bf423b6eb5f6 [term 1 LEADER]: Becoming Leader. State: Replica: 2fa77fc2d09d498993a3bf423b6eb5f6, State: Running, Role: LEADER
I20260928 02:14:08.242261 20920 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2fa77fc2d09d498993a3bf423b6eb5f6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2fa77fc2d09d498993a3bf423b6eb5f6" member_type: VOTER last_known_addr { host: "127.6.235.126" port: 43607 } }
I20260928 02:14:08.245739 20922 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2fa77fc2d09d498993a3bf423b6eb5f6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2fa77fc2d09d498993a3bf423b6eb5f6. Latest consensus state: current_term: 1 leader_uuid: "2fa77fc2d09d498993a3bf423b6eb5f6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2fa77fc2d09d498993a3bf423b6eb5f6" member_type: VOTER last_known_addr { host: "127.6.235.126" port: 43607 } } }
I20260928 02:14:08.246170 20922 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2fa77fc2d09d498993a3bf423b6eb5f6 [sys.catalog]: This master's current role is: LEADER
I20260928 02:14:08.246731 20921 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2fa77fc2d09d498993a3bf423b6eb5f6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2fa77fc2d09d498993a3bf423b6eb5f6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2fa77fc2d09d498993a3bf423b6eb5f6" member_type: VOTER last_known_addr { host: "127.6.235.126" port: 43607 } } }
I20260928 02:14:08.247365 20921 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2fa77fc2d09d498993a3bf423b6eb5f6 [sys.catalog]: This master's current role is: LEADER
W20260928 02:14:08.251212 20934 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 2fa77fc2d09d498993a3bf423b6eb5f6: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 02:14:08.251329 20934 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 02:14:08.251480 20935 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 02:14:08.253212 20935 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 02:14:08.261576 20935 catalog_manager.cc:1347] Generated new cluster ID: a4cb7f6e534647e89a298406b5f0c620
I20260928 02:14:08.261701 20935 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 02:14:08.277021 20935 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 02:14:08.279034 20935 catalog_manager.cc:1504] Loading token signing keys...
I20260928 02:14:08.290928 20935 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2fa77fc2d09d498993a3bf423b6eb5f6: Generated new TSK 0
I20260928 02:14:08.292109 20935 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 02:14:08.300700 7085 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskTlaUuc/build/asan/bin/kudu pid 20848
I20260928 02:14:08.302120 20910 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 02:14:08.395700 7085 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTlaUuc/build/asan/bin/kudu with pid 20848
2026-09-28T02:14:08Z chronyd exiting
[ OK ] SecurityITest.IPKICACertWebServerEndPoints (2012 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedIPKICert
Loading random data
Initializing database '/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790561432251112-7085-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 02:14:08 dist-test-slave-86gv krb5kdc[20943](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 02:14:08 dist-test-slave-86gv krb5kdc[20943](info): set up 2 sockets
Sep 28 02:14:08 dist-test-slave-86gv krb5kdc[20943](info): commencing operation
krb5kdc: starting...
W20260928 02:14:09.542057 7085 mini_kdc.cc:121] Time spent starting KDC: real 1.107s user 0.000s sys 0.025s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790561432251112-7085-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790561432251112-7085-0/krb5kdc/test-user.keytab.
Sep 28 02:14:09 dist-test-slave-86gv krb5kdc[20943](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561649, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T02:14:09Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T02:14:09Z Disabled control of system clock
WARNING: no policy specified for kudu/127.6.235.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.235.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.235.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790561432251112-7085-0/krb5kdc/kudu_127.6.235.126.keytab.
Entry for principal kudu/127.6.235.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790561432251112-7085-0/krb5kdc/kudu_127.6.235.126.keytab.
I20260928 02:14:09.793581 7085 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTlaUuc/build/asan/bin/kudu
/tmp/dist-test-taskTlaUuc/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790561432251112-7085-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790561432251112-7085-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790561432251112-7085-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790561432251112-7085-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.235.126:40053
--webserver_interface=127.6.235.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790561432251112-7085-0/krb5kdc/kudu_127.6.235.126.keytab
--principal=kudu/127.6.235.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.235.84:34573
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.235.126:40053
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790561432251112-7085-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790561432251112-7085-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790561432251112-7085-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:14:10.137646 20956 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:14:10.137961 20956 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:14:10.138159 20956 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:14:10.145645 20956 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 02:14:10.145752 20956 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:14:10.145843 20956 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:14:10.145917 20956 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 02:14:10.145980 20956 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 02:14:10.156519 20956 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.235.84:34573
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790561432251112-7085-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790561432251112-7085-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.235.126:40053
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790561432251112-7085-0/krb5kdc/kudu_127.6.235.126.keytab
--principal=kudu/127.6.235.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.235.126:40053
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790561432251112-7085-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.235.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790561432251112-7085-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 00:51:59 UTC on ccycloud.quasar-axgkwn.root.comops.site
build id 4676464
ASAN enabled
I20260928 02:14:10.158087 20956 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:14:10.159466 20956 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:14:10.171900 20962 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:14:10.171945 20961 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:14:10.171864 20964 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:14:10.178278 20956 server_base.cc:1034] running on GCE node
I20260928 02:14:10.179392 20956 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:14:10.181288 20956 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:14:10.182565 20956 hybrid_clock.cc:648] HybridClock initialized: now 1790561650182527 us; error 47 us; skew 500 ppm
I20260928 02:14:10.182997 20956 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 02:14:10 dist-test-slave-86gv krb5kdc[20943](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561650, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.235.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:14:10.186486 20956 init.cc:377] Logged in from keytab as kudu/127.6.235.126@KRBTEST.COM (short username kudu)
I20260928 02:14:10.188182 20956 webserver.cc:460] Webserver started at http://127.6.235.126:35131/ using document root <none> and password file <none>
I20260928 02:14:10.188997 20956 fs_manager.cc:359] Metadata directory not provided
I20260928 02:14:10.189112 20956 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:14:10.189388 20956 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:14:10.192471 20956 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790561432251112-7085-0/minicluster-data/master-0/data/instance:
uuid: "65164f35d42e4ab5a76865f2884ee008"
format_stamp: "Formatted at 2026-09-28 02:14:10 on dist-test-slave-86gv"
server_key: "004236509a5ac5a25bf575733d3d9620"
server_key_iv: "116136b2a19c5a524b3a72651389aeb2"
server_key_version: "encryptionkey@0"
I20260928 02:14:10.193284 20956 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790561432251112-7085-0/minicluster-data/master-0/wal/instance:
uuid: "65164f35d42e4ab5a76865f2884ee008"
format_stamp: "Formatted at 2026-09-28 02:14:10 on dist-test-slave-86gv"
server_key: "004236509a5ac5a25bf575733d3d9620"
server_key_iv: "116136b2a19c5a524b3a72651389aeb2"
server_key_version: "encryptionkey@0"
I20260928 02:14:10.198863 20956 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260928 02:14:10.202769 20971 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:14:10.204555 20956 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260928 02:14:10.204787 20956 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790561432251112-7085-0/minicluster-data/master-0/data,/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790561432251112-7085-0/minicluster-data/master-0/wal
uuid: "65164f35d42e4ab5a76865f2884ee008"
format_stamp: "Formatted at 2026-09-28 02:14:10 on dist-test-slave-86gv"
server_key: "004236509a5ac5a25bf575733d3d9620"
server_key_iv: "116136b2a19c5a524b3a72651389aeb2"
server_key_version: "encryptionkey@0"
I20260928 02:14:10.205055 20956 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790561432251112-7085-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790561432251112-7085-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790561432251112-7085-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:14:10.222682 20956 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:14:10.227087 20956 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:14:10.227427 20956 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:14:10.247632 20956 rpc_server.cc:307] RPC server started. Bound to: 127.6.235.126:40053
I20260928 02:14:10.247735 21022 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.235.126:40053 every 8 connection(s)
I20260928 02:14:10.249161 20956 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790561432251112-7085-0/minicluster-data/master-0/data/info.pb
I20260928 02:14:10.253247 21023 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:14:10.258201 7085 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTlaUuc/build/asan/bin/kudu as pid 20956
I20260928 02:14:10.258430 7085 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790561432251112-7085-0/minicluster-data/master-0/wal/instance
I20260928 02:14:10.259325 7085 external_mini_cluster.cc:1428] Setting key 2a681c7ab070ef8871df5f591717bc0a
I20260928 02:14:10.265880 21023 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 65164f35d42e4ab5a76865f2884ee008: Bootstrap starting.
I20260928 02:14:10.270026 21023 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 65164f35d42e4ab5a76865f2884ee008: Neither blocks nor log segments found. Creating new log.
I20260928 02:14:10.271432 21023 log.cc:824] T 00000000000000000000000000000000 P 65164f35d42e4ab5a76865f2884ee008: Log is configured to *not* fsync() on all Append() calls
I20260928 02:14:10.275239 21023 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 65164f35d42e4ab5a76865f2884ee008: No bootstrap required, opened a new log
Sep 28 02:14:10 dist-test-slave-86gv krb5kdc[20943](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561649, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.235.126@KRBTEST.COM
I20260928 02:14:10.283577 21023 raft_consensus.cc:348] T 00000000000000000000000000000000 P 65164f35d42e4ab5a76865f2884ee008 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "65164f35d42e4ab5a76865f2884ee008" member_type: VOTER last_known_addr { host: "127.6.235.126" port: 40053 } }
I20260928 02:14:10.284016 21023 raft_consensus.cc:374] T 00000000000000000000000000000000 P 65164f35d42e4ab5a76865f2884ee008 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:14:10.284148 21023 raft_consensus.cc:729] T 00000000000000000000000000000000 P 65164f35d42e4ab5a76865f2884ee008 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 65164f35d42e4ab5a76865f2884ee008, State: Initialized, Role: FOLLOWER
I20260928 02:14:10.284940 21023 consensus_queue.cc:260] T 00000000000000000000000000000000 P 65164f35d42e4ab5a76865f2884ee008 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "65164f35d42e4ab5a76865f2884ee008" member_type: VOTER last_known_addr { host: "127.6.235.126" port: 40053 } }
I20260928 02:14:10.285249 21023 raft_consensus.cc:388] T 00000000000000000000000000000000 P 65164f35d42e4ab5a76865f2884ee008 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 02:14:10.285414 21023 raft_consensus.cc:482] T 00000000000000000000000000000000 P 65164f35d42e4ab5a76865f2884ee008 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 02:14:10.285645 21023 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 65164f35d42e4ab5a76865f2884ee008 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:14:10.286319 21026 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:14:10.261867 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35482 (local address 127.6.235.126:40053)
0928 02:14:10.262997 (+ 1130us) server_negotiation.cc:206] Beginning negotiation
0928 02:14:10.263016 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0928 02:14:10.263072 (+ 56us) server_negotiation.cc:407] Connection header received
0928 02:14:10.265352 (+ 2280us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:14:10.265380 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:14:10.265896 (+ 516us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:14:10.266358 (+ 462us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:14:10.268280 (+ 1922us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:14:10.269859 (+ 1579us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:14:10.271969 (+ 2110us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:14:10.273194 (+ 1225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:14:10.273355 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:14:10.278321 (+ 4966us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:14:10.278329 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:14:10.278345 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:14:10.278391 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:14:10.281701 (+ 3310us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:14:10.282501 (+ 800us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:14:10.282508 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:14:10.282515 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:14:10.282704 (+ 189us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:14:10.283113 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:14:10.283118 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:14:10.283123 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:14:10.283713 (+ 590us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:14:10.284053 (+ 340us) server_negotiation.cc:1035] Waiting for connection context
0928 02:14:10.284506 (+ 453us) server_negotiation.cc:299] Negotiation successful
0928 02:14:10.285039 (+ 533us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":823,"thread_start_us":552,"threads_started":1}
I20260928 02:14:10.288642 21023 raft_consensus.cc:504] T 00000000000000000000000000000000 P 65164f35d42e4ab5a76865f2884ee008 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "65164f35d42e4ab5a76865f2884ee008" member_type: VOTER last_known_addr { host: "127.6.235.126" port: 40053 } }
I20260928 02:14:10.289247 21023 leader_election.cc:302] T 00000000000000000000000000000000 P 65164f35d42e4ab5a76865f2884ee008 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 65164f35d42e4ab5a76865f2884ee008; no voters:
I20260928 02:14:10.290002 21023 leader_election.cc:288] T 00000000000000000000000000000000 P 65164f35d42e4ab5a76865f2884ee008 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 02:14:10.290148 21028 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 65164f35d42e4ab5a76865f2884ee008 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:14:10.290907 21028 raft_consensus.cc:686] T 00000000000000000000000000000000 P 65164f35d42e4ab5a76865f2884ee008 [term 1 LEADER]: Becoming Leader. State: Replica: 65164f35d42e4ab5a76865f2884ee008, State: Running, Role: LEADER
I20260928 02:14:10.291473 21028 consensus_queue.cc:237] T 00000000000000000000000000000000 P 65164f35d42e4ab5a76865f2884ee008 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "65164f35d42e4ab5a76865f2884ee008" member_type: VOTER last_known_addr { host: "127.6.235.126" port: 40053 } }
I20260928 02:14:10.292289 21023 sys_catalog.cc:564] T 00000000000000000000000000000000 P 65164f35d42e4ab5a76865f2884ee008 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 02:14:10.294564 21030 sys_catalog.cc:455] T 00000000000000000000000000000000 P 65164f35d42e4ab5a76865f2884ee008 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 65164f35d42e4ab5a76865f2884ee008. Latest consensus state: current_term: 1 leader_uuid: "65164f35d42e4ab5a76865f2884ee008" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "65164f35d42e4ab5a76865f2884ee008" member_type: VOTER last_known_addr { host: "127.6.235.126" port: 40053 } } }
I20260928 02:14:10.294648 21029 sys_catalog.cc:455] T 00000000000000000000000000000000 P 65164f35d42e4ab5a76865f2884ee008 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "65164f35d42e4ab5a76865f2884ee008" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "65164f35d42e4ab5a76865f2884ee008" member_type: VOTER last_known_addr { host: "127.6.235.126" port: 40053 } } }
I20260928 02:14:10.294961 21030 sys_catalog.cc:458] T 00000000000000000000000000000000 P 65164f35d42e4ab5a76865f2884ee008 [sys.catalog]: This master's current role is: LEADER
I20260928 02:14:10.295096 21029 sys_catalog.cc:458] T 00000000000000000000000000000000 P 65164f35d42e4ab5a76865f2884ee008 [sys.catalog]: This master's current role is: LEADER
I20260928 02:14:10.296816 21034 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 02:14:10.302335 21034 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 02:14:10.312572 21034 catalog_manager.cc:1347] Generated new cluster ID: 6dfacd5e0cbd4b6fb1ce99f299c840f2
I20260928 02:14:10.312722 21034 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 02:14:10.351898 21034 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 02:14:10.353410 21034 catalog_manager.cc:1504] Loading token signing keys...
I20260928 02:14:10.360353 21034 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 65164f35d42e4ab5a76865f2884ee008: Generated new TSK 0
I20260928 02:14:10.361373 21034 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.6.235.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.235.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.235.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790561432251112-7085-0/krb5kdc/kudu_127.6.235.65.keytab.
Entry for principal kudu/127.6.235.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790561432251112-7085-0/krb5kdc/kudu_127.6.235.65.keytab.
I20260928 02:14:10.437701 7085 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTlaUuc/build/asan/bin/kudu
/tmp/dist-test-taskTlaUuc/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790561432251112-7085-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790561432251112-7085-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790561432251112-7085-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790561432251112-7085-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.235.65:0
--local_ip_for_outbound_sockets=127.6.235.65
--webserver_interface=127.6.235.65
--webserver_port=0
--tserver_master_addrs=127.6.235.126:40053
--keytab_file=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790561432251112-7085-0/krb5kdc/kudu_127.6.235.65.keytab
--principal=kudu/127.6.235.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.235.84:34573
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790561432251112-7085-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790561432251112-7085-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790561432251112-7085-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:14:10.819465 21049 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:14:10.819773 21049 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:14:10.819924 21049 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:14:10.827607 21049 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:14:10.827723 21049 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:14:10.827893 21049 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.235.65
I20260928 02:14:10.838315 21049 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.235.84:34573
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790561432251112-7085-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790561432251112-7085-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790561432251112-7085-0/krb5kdc/kudu_127.6.235.65.keytab
--principal=kudu/127.6.235.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.235.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790561432251112-7085-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.6.235.65
--webserver_port=0
--tserver_master_addrs=127.6.235.126:40053
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.235.65
--log_dir=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790561432251112-7085-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 00:51:59 UTC on ccycloud.quasar-axgkwn.root.comops.site
build id 4676464
ASAN enabled
I20260928 02:14:10.839720 21049 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:14:10.841178 21049 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:14:10.855302 21055 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:14:10.855327 21054 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:14:10.856181 21057 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:14:10.856158 21049 server_base.cc:1034] running on GCE node
I20260928 02:14:10.856947 21049 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:14:10.857996 21049 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:14:10.859297 21049 hybrid_clock.cc:648] HybridClock initialized: now 1790561650859203 us; error 62 us; skew 500 ppm
I20260928 02:14:10.859848 21049 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 02:14:10 dist-test-slave-86gv krb5kdc[20943](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561650, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.235.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:14:10.863391 21049 init.cc:377] Logged in from keytab as kudu/127.6.235.65@KRBTEST.COM (short username kudu)
I20260928 02:14:10.865114 21049 webserver.cc:460] Webserver started at http://127.6.235.65:41415/ using document root <none> and password file <none>
I20260928 02:14:10.865895 21049 fs_manager.cc:359] Metadata directory not provided
I20260928 02:14:10.866061 21049 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:14:10.866483 21049 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:14:10.869735 21049 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790561432251112-7085-0/minicluster-data/ts-0/data/instance:
uuid: "37070660923a4173b9bfe2c5b3abc0c4"
format_stamp: "Formatted at 2026-09-28 02:14:10 on dist-test-slave-86gv"
server_key: "1b06c11186b1b991199074c31887352b"
server_key_iv: "1f600e3577ba6b34682b7fb5036eb7df"
server_key_version: "encryptionkey@0"
I20260928 02:14:10.870579 21049 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790561432251112-7085-0/minicluster-data/ts-0/wal/instance:
uuid: "37070660923a4173b9bfe2c5b3abc0c4"
format_stamp: "Formatted at 2026-09-28 02:14:10 on dist-test-slave-86gv"
server_key: "1b06c11186b1b991199074c31887352b"
server_key_iv: "1f600e3577ba6b34682b7fb5036eb7df"
server_key_version: "encryptionkey@0"
I20260928 02:14:10.876537 21049 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.001s
I20260928 02:14:10.880509 21064 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:14:10.882036 21049 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 02:14:10.882227 21049 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790561432251112-7085-0/minicluster-data/ts-0/data,/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790561432251112-7085-0/minicluster-data/ts-0/wal
uuid: "37070660923a4173b9bfe2c5b3abc0c4"
format_stamp: "Formatted at 2026-09-28 02:14:10 on dist-test-slave-86gv"
server_key: "1b06c11186b1b991199074c31887352b"
server_key_iv: "1f600e3577ba6b34682b7fb5036eb7df"
server_key_version: "encryptionkey@0"
I20260928 02:14:10.882447 21049 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790561432251112-7085-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790561432251112-7085-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790561432251112-7085-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:14:10.901918 21049 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:14:10.906268 21049 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:14:10.906616 21049 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:14:10.907657 21049 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:14:10.909219 21049 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:14:10.909322 21049 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:14:10.909477 21049 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:14:10.909551 21049 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:14:10.944823 21049 rpc_server.cc:307] RPC server started. Bound to: 127.6.235.65:37261
I20260928 02:14:10.944859 21176 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.235.65:37261 every 8 connection(s)
I20260928 02:14:10.946297 21049 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790561432251112-7085-0/minicluster-data/ts-0/data/info.pb
I20260928 02:14:10.949584 7085 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTlaUuc/build/asan/bin/kudu as pid 21049
I20260928 02:14:10.949877 7085 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790561432251112-7085-0/minicluster-data/ts-0/wal/instance
I20260928 02:14:10.950565 7085 external_mini_cluster.cc:1428] Setting key 312ceb3bac9b93bb33ba5ee932ad1f01
Sep 28 02:14:10 dist-test-slave-86gv krb5kdc[20943](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561650, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.235.65@KRBTEST.COM for kudu/127.6.235.126@KRBTEST.COM
I20260928 02:14:10.968813 21179 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:14:10.948996 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.235.65:60825 (local address 127.6.235.126:40053)
0928 02:14:10.949559 (+ 563us) server_negotiation.cc:206] Beginning negotiation
0928 02:14:10.949568 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 02:14:10.951004 (+ 1436us) server_negotiation.cc:407] Connection header received
0928 02:14:10.953236 (+ 2232us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:14:10.953241 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:14:10.953326 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:14:10.953451 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:14:10.955024 (+ 1573us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:14:10.955744 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:14:10.957402 (+ 1658us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:14:10.957804 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:14:10.957961 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:14:10.961949 (+ 3988us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:14:10.961954 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:14:10.961957 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:14:10.961993 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:14:10.964674 (+ 2681us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:14:10.965781 (+ 1107us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:14:10.965785 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:14:10.965788 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:14:10.965921 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:14:10.966509 (+ 588us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:14:10.966516 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:14:10.966522 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:14:10.966913 (+ 391us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:14:10.967077 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0928 02:14:10.968023 (+ 946us) server_negotiation.cc:299] Negotiation successful
0928 02:14:10.968302 (+ 279us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":403,"thread_start_us":253,"threads_started":1}
I20260928 02:14:10.969483 21180 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:14:10.949638 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.235.126:40053 (local address 127.6.235.65:60825)
0928 02:14:10.950535 (+ 897us) negotiation.cc:107] Waiting for socket to connect
0928 02:14:10.950590 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0928 02:14:10.952954 (+ 2364us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:14:10.953721 (+ 767us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:14:10.953730 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:14:10.954253 (+ 523us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:14:10.954842 (+ 589us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:14:10.954859 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:14:10.955919 (+ 1060us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:14:10.955924 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:14:10.957244 (+ 1320us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:14:10.957257 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:14:10.958048 (+ 791us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:14:10.958056 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:14:10.958300 (+ 244us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:14:10.959078 (+ 778us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:14:10.959105 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:14:10.961670 (+ 2565us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:14:10.964837 (+ 3167us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:14:10.964850 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:14:10.964872 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:14:10.965605 (+ 733us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:14:10.966051 (+ 446us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:14:10.966060 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:14:10.966067 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:14:10.966327 (+ 260us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:14:10.967054 (+ 727us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:14:10.967061 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:14:10.967581 (+ 520us) client_negotiation.cc:769] Sending connection context
0928 02:14:10.967849 (+ 268us) client_negotiation.cc:240] Negotiation successful
0928 02:14:10.968275 (+ 426us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":597,"thread_start_us":363,"threads_started":1}
I20260928 02:14:10.971164 21177 heartbeater.cc:347] Connected to a master server at 127.6.235.126:40053
I20260928 02:14:10.971550 21177 heartbeater.cc:464] Registering TS with master...
I20260928 02:14:10.972599 21177 heartbeater.cc:511] Master 127.6.235.126:40053 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.6.235.66@KRBTEST.COM; defaulting to no policy
I20260928 02:14:10.975148 20988 ts_manager.cc:194] Registered new tserver with Master: 37070660923a4173b9bfe2c5b3abc0c4 (127.6.235.65:37261)
I20260928 02:14:10.976792 20988 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.235.65@KRBTEST.COM'} at 127.6.235.65:60825
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.235.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.235.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790561432251112-7085-0/krb5kdc/kudu_127.6.235.66.keytab.
Entry for principal kudu/127.6.235.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790561432251112-7085-0/krb5kdc/kudu_127.6.235.66.keytab.
I20260928 02:14:11.009648 7085 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTlaUuc/build/asan/bin/kudu
/tmp/dist-test-taskTlaUuc/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790561432251112-7085-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790561432251112-7085-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790561432251112-7085-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790561432251112-7085-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.235.66:0
--local_ip_for_outbound_sockets=127.6.235.66
--webserver_interface=127.6.235.66
--webserver_port=0
--tserver_master_addrs=127.6.235.126:40053
--keytab_file=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790561432251112-7085-0/krb5kdc/kudu_127.6.235.66.keytab
--principal=kudu/127.6.235.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.235.84:34573
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790561432251112-7085-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790561432251112-7085-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790561432251112-7085-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:14:11.350629 21183 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:14:11.351014 21183 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:14:11.351197 21183 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:14:11.358613 21183 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:14:11.358798 21183 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:14:11.359005 21183 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.235.66
I20260928 02:14:11.369108 21183 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.235.84:34573
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790561432251112-7085-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790561432251112-7085-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790561432251112-7085-0/krb5kdc/kudu_127.6.235.66.keytab
--principal=kudu/127.6.235.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.235.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790561432251112-7085-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.6.235.66
--webserver_port=0
--tserver_master_addrs=127.6.235.126:40053
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.235.66
--log_dir=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790561432251112-7085-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 00:51:59 UTC on ccycloud.quasar-axgkwn.root.comops.site
build id 4676464
ASAN enabled
I20260928 02:14:11.370512 21183 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:14:11.371907 21183 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:14:11.385910 21189 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:14:11.385998 21188 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:14:11.385954 21191 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:14:11.386902 21183 server_base.cc:1034] running on GCE node
I20260928 02:14:11.387588 21183 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:14:11.388609 21183 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:14:11.389845 21183 hybrid_clock.cc:648] HybridClock initialized: now 1790561651389766 us; error 75 us; skew 500 ppm
I20260928 02:14:11.390185 21183 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 02:14:11 dist-test-slave-86gv krb5kdc[20943](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561651, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.235.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:14:11.393540 21183 init.cc:377] Logged in from keytab as kudu/127.6.235.66@KRBTEST.COM (short username kudu)
I20260928 02:14:11.395011 21183 webserver.cc:460] Webserver started at http://127.6.235.66:45689/ using document root <none> and password file <none>
I20260928 02:14:11.395691 21183 fs_manager.cc:359] Metadata directory not provided
I20260928 02:14:11.395787 21183 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:14:11.396026 21183 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:14:11.399096 21183 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790561432251112-7085-0/minicluster-data/ts-1/data/instance:
uuid: "f820529e7c8e4f2f8341f373b1def2d0"
format_stamp: "Formatted at 2026-09-28 02:14:11 on dist-test-slave-86gv"
server_key: "2121432c3e1f641baf637d3823de8d3d"
server_key_iv: "659dd1f1aa6458b3c06d8512e968d2c9"
server_key_version: "encryptionkey@0"
I20260928 02:14:11.399884 21183 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790561432251112-7085-0/minicluster-data/ts-1/wal/instance:
uuid: "f820529e7c8e4f2f8341f373b1def2d0"
format_stamp: "Formatted at 2026-09-28 02:14:11 on dist-test-slave-86gv"
server_key: "2121432c3e1f641baf637d3823de8d3d"
server_key_iv: "659dd1f1aa6458b3c06d8512e968d2c9"
server_key_version: "encryptionkey@0"
I20260928 02:14:11.405290 21183 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.005s
I20260928 02:14:11.409005 21198 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:14:11.410570 21183 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260928 02:14:11.410725 21183 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790561432251112-7085-0/minicluster-data/ts-1/data,/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790561432251112-7085-0/minicluster-data/ts-1/wal
uuid: "f820529e7c8e4f2f8341f373b1def2d0"
format_stamp: "Formatted at 2026-09-28 02:14:11 on dist-test-slave-86gv"
server_key: "2121432c3e1f641baf637d3823de8d3d"
server_key_iv: "659dd1f1aa6458b3c06d8512e968d2c9"
server_key_version: "encryptionkey@0"
I20260928 02:14:11.410918 21183 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790561432251112-7085-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790561432251112-7085-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790561432251112-7085-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:14:11.443743 21183 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:14:11.448139 21183 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:14:11.448392 21183 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:14:11.449398 21183 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:14:11.450910 21183 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:14:11.450989 21183 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:14:11.451082 21183 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:14:11.451128 21183 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:14:11.485138 21183 rpc_server.cc:307] RPC server started. Bound to: 127.6.235.66:36697
I20260928 02:14:11.485189 21310 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.235.66:36697 every 8 connection(s)
I20260928 02:14:11.486510 21183 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790561432251112-7085-0/minicluster-data/ts-1/data/info.pb
I20260928 02:14:11.494844 7085 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTlaUuc/build/asan/bin/kudu as pid 21183
I20260928 02:14:11.495079 7085 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790561432251112-7085-0/minicluster-data/ts-1/wal/instance
I20260928 02:14:11.495615 7085 external_mini_cluster.cc:1428] Setting key 0b0b690614354e318549571209f4a717
Sep 28 02:14:11 dist-test-slave-86gv krb5kdc[20943](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561651, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.235.66@KRBTEST.COM for kudu/127.6.235.126@KRBTEST.COM
I20260928 02:14:11.518368 21313 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:14:11.489272 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.235.66:57389 (local address 127.6.235.126:40053)
0928 02:14:11.489738 (+ 466us) server_negotiation.cc:206] Beginning negotiation
0928 02:14:11.489743 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 02:14:11.490816 (+ 1073us) server_negotiation.cc:407] Connection header received
0928 02:14:11.492705 (+ 1889us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:14:11.492709 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:14:11.492782 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:14:11.492885 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:14:11.495087 (+ 2202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:14:11.495982 (+ 895us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:14:11.498110 (+ 2128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:14:11.498546 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:14:11.498691 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:14:11.510359 (+ 11668us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:14:11.510369 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:14:11.510377 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:14:11.510430 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:14:11.514344 (+ 3914us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:14:11.515221 (+ 877us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:14:11.515229 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:14:11.515235 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:14:11.515428 (+ 193us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:14:11.515893 (+ 465us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:14:11.515900 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:14:11.515906 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:14:11.516337 (+ 431us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:14:11.516501 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0928 02:14:11.517547 (+ 1046us) server_negotiation.cc:299] Negotiation successful
0928 02:14:11.517838 (+ 291us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":350,"thread_start_us":242,"threads_started":1}
I20260928 02:14:11.519124 21314 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:14:11.489942 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.235.126:40053 (local address 127.6.235.66:57389)
0928 02:14:11.490638 (+ 696us) negotiation.cc:107] Waiting for socket to connect
0928 02:14:11.490681 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0928 02:14:11.492405 (+ 1724us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:14:11.493178 (+ 773us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:14:11.493188 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:14:11.493739 (+ 551us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:14:11.494403 (+ 664us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:14:11.494422 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:14:11.496174 (+ 1752us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:14:11.496178 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:14:11.497951 (+ 1773us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:14:11.497965 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:14:11.498799 (+ 834us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:14:11.498805 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:14:11.499069 (+ 264us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:14:11.506955 (+ 7886us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:14:11.506993 (+ 38us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:14:11.510006 (+ 3013us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:14:11.514533 (+ 4527us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:14:11.514543 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:14:11.514559 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:14:11.515055 (+ 496us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:14:11.515553 (+ 498us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:14:11.515557 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:14:11.515559 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:14:11.515760 (+ 201us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:14:11.516504 (+ 744us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:14:11.516511 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:14:11.517094 (+ 583us) client_negotiation.cc:769] Sending connection context
0928 02:14:11.517376 (+ 282us) client_negotiation.cc:240] Negotiation successful
0928 02:14:11.517869 (+ 493us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":444,"thread_start_us":275,"threads_started":1}
WARNING: no policy specified for kudu/127.6.235.67@KRBTEST.COM; defaulting to no policy
I20260928 02:14:11.520848 21311 heartbeater.cc:347] Connected to a master server at 127.6.235.126:40053
I20260928 02:14:11.521253 21311 heartbeater.cc:464] Registering TS with master...
I20260928 02:14:11.522042 21311 heartbeater.cc:511] Master 127.6.235.126:40053 requested a full tablet report, sending...
I20260928 02:14:11.523548 20988 ts_manager.cc:194] Registered new tserver with Master: f820529e7c8e4f2f8341f373b1def2d0 (127.6.235.66:36697)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.235.67@KRBTEST.COM" created.
I20260928 02:14:11.524725 20988 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.235.66@KRBTEST.COM'} at 127.6.235.66:57389
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.235.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790561432251112-7085-0/krb5kdc/kudu_127.6.235.67.keytab.
Entry for principal kudu/127.6.235.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790561432251112-7085-0/krb5kdc/kudu_127.6.235.67.keytab.
I20260928 02:14:11.553594 7085 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTlaUuc/build/asan/bin/kudu
/tmp/dist-test-taskTlaUuc/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790561432251112-7085-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790561432251112-7085-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790561432251112-7085-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790561432251112-7085-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.235.67:0
--local_ip_for_outbound_sockets=127.6.235.67
--webserver_interface=127.6.235.67
--webserver_port=0
--tserver_master_addrs=127.6.235.126:40053
--keytab_file=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790561432251112-7085-0/krb5kdc/kudu_127.6.235.67.keytab
--principal=kudu/127.6.235.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.235.84:34573
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790561432251112-7085-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790561432251112-7085-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790561432251112-7085-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:14:11.907866 21317 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:14:11.908238 21317 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:14:11.908439 21317 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:14:11.917323 21317 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:14:11.917465 21317 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:14:11.917656 21317 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.235.67
I20260928 02:14:11.928467 21317 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.235.84:34573
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790561432251112-7085-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790561432251112-7085-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790561432251112-7085-0/krb5kdc/kudu_127.6.235.67.keytab
--principal=kudu/127.6.235.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.235.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790561432251112-7085-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.6.235.67
--webserver_port=0
--tserver_master_addrs=127.6.235.126:40053
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.235.67
--log_dir=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790561432251112-7085-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 00:51:59 UTC on ccycloud.quasar-axgkwn.root.comops.site
build id 4676464
ASAN enabled
I20260928 02:14:11.929898 21317 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:14:11.931281 21317 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:14:11.944916 21323 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:14:11.945081 21325 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:14:11.945640 21322 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:14:11.946031 21317 server_base.cc:1034] running on GCE node
I20260928 02:14:11.946702 21317 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:14:11.947764 21317 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:14:11.949038 21317 hybrid_clock.cc:648] HybridClock initialized: now 1790561651948977 us; error 51 us; skew 500 ppm
I20260928 02:14:11.949419 21317 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 02:14:11 dist-test-slave-86gv krb5kdc[20943](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561651, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.235.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:14:11.952808 21317 init.cc:377] Logged in from keytab as kudu/127.6.235.67@KRBTEST.COM (short username kudu)
I20260928 02:14:11.954306 21317 webserver.cc:460] Webserver started at http://127.6.235.67:40207/ using document root <none> and password file <none>
I20260928 02:14:11.955052 21317 fs_manager.cc:359] Metadata directory not provided
I20260928 02:14:11.955197 21317 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:14:11.955513 21317 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:14:11.958560 21317 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790561432251112-7085-0/minicluster-data/ts-2/data/instance:
uuid: "f11b50d0630c4d91be9b17973fcb0230"
format_stamp: "Formatted at 2026-09-28 02:14:11 on dist-test-slave-86gv"
server_key: "51fe6d056371e4b2efa727b487ae6e62"
server_key_iv: "32e2a67cf03cccfcde49d183927eee55"
server_key_version: "encryptionkey@0"
I20260928 02:14:11.959375 21317 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790561432251112-7085-0/minicluster-data/ts-2/wal/instance:
uuid: "f11b50d0630c4d91be9b17973fcb0230"
format_stamp: "Formatted at 2026-09-28 02:14:11 on dist-test-slave-86gv"
server_key: "51fe6d056371e4b2efa727b487ae6e62"
server_key_iv: "32e2a67cf03cccfcde49d183927eee55"
server_key_version: "encryptionkey@0"
I20260928 02:14:11.964651 21317 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260928 02:14:11.968345 21332 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:14:11.969911 21317 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 02:14:11.970100 21317 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790561432251112-7085-0/minicluster-data/ts-2/data,/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790561432251112-7085-0/minicluster-data/ts-2/wal
uuid: "f11b50d0630c4d91be9b17973fcb0230"
format_stamp: "Formatted at 2026-09-28 02:14:11 on dist-test-slave-86gv"
server_key: "51fe6d056371e4b2efa727b487ae6e62"
server_key_iv: "32e2a67cf03cccfcde49d183927eee55"
server_key_version: "encryptionkey@0"
I20260928 02:14:11.970324 21317 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790561432251112-7085-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790561432251112-7085-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790561432251112-7085-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:14:11.980129 21177 heartbeater.cc:503] Master 127.6.235.126:40053 was elected leader, sending a full tablet report...
I20260928 02:14:11.991286 21317 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:14:11.996160 21317 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:14:11.996497 21317 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:14:11.997772 21317 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:14:11.999334 21317 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:14:11.999451 21317 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:14:11.999603 21317 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:14:11.999675 21317 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:14:12.033589 21317 rpc_server.cc:307] RPC server started. Bound to: 127.6.235.67:41399
I20260928 02:14:12.033634 21444 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.235.67:41399 every 8 connection(s)
I20260928 02:14:12.034907 21317 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790561432251112-7085-0/minicluster-data/ts-2/data/info.pb
I20260928 02:14:12.038093 7085 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTlaUuc/build/asan/bin/kudu as pid 21317
I20260928 02:14:12.038286 7085 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790561432251112-7085-0/minicluster-data/ts-2/wal/instance
I20260928 02:14:12.038853 7085 external_mini_cluster.cc:1428] Setting key 7bd4472f495bce98c58d0d9ead844448
Sep 28 02:14:12 dist-test-slave-86gv krb5kdc[20943](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561651, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.235.67@KRBTEST.COM for kudu/127.6.235.126@KRBTEST.COM
I20260928 02:14:12.056259 21447 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:14:12.037679 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.235.67:38493 (local address 127.6.235.126:40053)
0928 02:14:12.038273 (+ 594us) server_negotiation.cc:206] Beginning negotiation
0928 02:14:12.038283 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 02:14:12.039514 (+ 1231us) server_negotiation.cc:407] Connection header received
0928 02:14:12.041662 (+ 2148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:14:12.041668 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:14:12.041764 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:14:12.041884 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:14:12.043503 (+ 1619us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:14:12.044259 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:14:12.045925 (+ 1666us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:14:12.046330 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:14:12.046474 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:14:12.050230 (+ 3756us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:14:12.050235 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:14:12.050239 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:14:12.050274 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:14:12.053072 (+ 2798us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:14:12.053868 (+ 796us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:14:12.053873 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:14:12.053877 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:14:12.054013 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:14:12.054473 (+ 460us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:14:12.054477 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:14:12.054479 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:14:12.054753 (+ 274us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:14:12.054925 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0928 02:14:12.055640 (+ 715us) server_negotiation.cc:299] Negotiation successful
0928 02:14:12.055863 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":442,"thread_start_us":294,"threads_started":1}
I20260928 02:14:12.057387 21448 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:14:12.038425 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.235.126:40053 (local address 127.6.235.67:38493)
0928 02:14:12.039308 (+ 883us) negotiation.cc:107] Waiting for socket to connect
0928 02:14:12.039374 (+ 66us) client_negotiation.cc:174] Beginning negotiation
0928 02:14:12.041384 (+ 2010us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:14:12.042154 (+ 770us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:14:12.042164 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:14:12.042699 (+ 535us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:14:12.043316 (+ 617us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:14:12.043333 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:14:12.044412 (+ 1079us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:14:12.044417 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:14:12.045774 (+ 1357us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:14:12.045786 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:14:12.046571 (+ 785us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:14:12.046580 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:14:12.046821 (+ 241us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:14:12.047568 (+ 747us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:14:12.047600 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:14:12.049995 (+ 2395us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:14:12.053224 (+ 3229us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:14:12.053234 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:14:12.053259 (+ 25us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:14:12.053732 (+ 473us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:14:12.054128 (+ 396us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:14:12.054133 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:14:12.054136 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:14:12.054336 (+ 200us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:14:12.054887 (+ 551us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:14:12.054895 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:14:12.055271 (+ 376us) client_negotiation.cc:769] Sending connection context
0928 02:14:12.055557 (+ 286us) client_negotiation.cc:240] Negotiation successful
0928 02:14:12.056131 (+ 574us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":563,"thread_start_us":310,"threads_started":1}
I20260928 02:14:12.058804 21445 heartbeater.cc:347] Connected to a master server at 127.6.235.126:40053
I20260928 02:14:12.059146 21445 heartbeater.cc:464] Registering TS with master...
I20260928 02:14:12.059947 21445 heartbeater.cc:511] Master 127.6.235.126:40053 requested a full tablet report, sending...
I20260928 02:14:12.061661 20988 ts_manager.cc:194] Registered new tserver with Master: f11b50d0630c4d91be9b17973fcb0230 (127.6.235.67:41399)
I20260928 02:14:12.062498 20988 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.235.67@KRBTEST.COM'} at 127.6.235.67:38493
I20260928 02:14:12.066955 7085 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 02:14:12.084254 21447 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:14:12.071466 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35006 (local address 127.6.235.126:40053)
0928 02:14:12.071661 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0928 02:14:12.071667 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 02:14:12.072192 (+ 525us) server_negotiation.cc:407] Connection header received
0928 02:14:12.072331 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:14:12.072334 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:14:12.072400 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:14:12.072490 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:14:12.073901 (+ 1411us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:14:12.074697 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:14:12.076080 (+ 1383us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:14:12.076515 (+ 435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:14:12.076627 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:14:12.079145 (+ 2518us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:14:12.079149 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:14:12.079153 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:14:12.079187 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:14:12.081980 (+ 2793us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:14:12.082641 (+ 661us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:14:12.082645 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:14:12.082647 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:14:12.082768 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:14:12.083094 (+ 326us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:14:12.083097 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:14:12.083099 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:14:12.083407 (+ 308us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:14:12.083627 (+ 220us) server_negotiation.cc:1035] Waiting for connection context
0928 02:14:12.083854 (+ 227us) server_negotiation.cc:299] Negotiation successful
0928 02:14:12.084027 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260928 02:14:12.088497 20988 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35006:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 02:14:12.095630 20988 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 02:14:12.120083 21459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:14:12.108978 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48296 (local address 127.6.235.66:36697)
0928 02:14:12.109702 (+ 724us) server_negotiation.cc:206] Beginning negotiation
0928 02:14:12.109711 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 02:14:12.110270 (+ 559us) server_negotiation.cc:407] Connection header received
0928 02:14:12.110690 (+ 420us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:14:12.110697 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:14:12.110922 (+ 225us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:14:12.111136 (+ 214us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:14:12.112289 (+ 1153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:14:12.113852 (+ 1563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:14:12.116788 (+ 2936us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:14:12.118370 (+ 1582us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:14:12.118527 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:14:12.118724 (+ 197us) server_negotiation.cc:1035] Waiting for connection context
0928 02:14:12.119503 (+ 779us) server_negotiation.cc:299] Negotiation successful
0928 02:14:12.119702 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":541,"thread_start_us":376,"threads_started":1}
I20260928 02:14:12.120585 21461 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:14:12.109795 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.235.67:41399 (local address 127.0.0.1:58842)
0928 02:14:12.110781 (+ 986us) negotiation.cc:107] Waiting for socket to connect
0928 02:14:12.110810 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0928 02:14:12.110959 (+ 149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:14:12.112176 (+ 1217us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:14:12.112183 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:14:12.112211 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:14:12.112575 (+ 364us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:14:12.112587 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:14:12.114611 (+ 2024us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:14:12.114616 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:14:12.116777 (+ 2161us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:14:12.116791 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:14:12.118906 (+ 2115us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:14:12.118912 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:14:12.119165 (+ 253us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:14:12.119274 (+ 109us) client_negotiation.cc:769] Sending connection context
0928 02:14:12.119416 (+ 142us) client_negotiation.cc:240] Negotiation successful
0928 02:14:12.119710 (+ 294us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":855,"mutex_wait_us":135,"thread_start_us":663,"threads_started":1}
I20260928 02:14:12.120613 21458 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:14:12.108726 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.235.66:36697 (local address 127.0.0.1:48296)
0928 02:14:12.110133 (+ 1407us) negotiation.cc:107] Waiting for socket to connect
0928 02:14:12.110171 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0928 02:14:12.110421 (+ 250us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:14:12.111538 (+ 1117us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:14:12.111552 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:14:12.111585 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:14:12.112084 (+ 499us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:14:12.112099 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:14:12.114085 (+ 1986us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:14:12.114093 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:14:12.116626 (+ 2533us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:14:12.116639 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:14:12.118893 (+ 2254us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:14:12.118900 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:14:12.119165 (+ 265us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:14:12.119274 (+ 109us) client_negotiation.cc:769] Sending connection context
0928 02:14:12.119416 (+ 142us) client_negotiation.cc:240] Negotiation successful
0928 02:14:12.119710 (+ 294us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1176,"mutex_wait_us":184,"thread_start_us":1009,"threads_started":1}
I20260928 02:14:12.120813 21460 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:14:12.109406 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58842 (local address 127.6.235.67:41399)
0928 02:14:12.110265 (+ 859us) server_negotiation.cc:206] Beginning negotiation
0928 02:14:12.110276 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 02:14:12.110928 (+ 652us) server_negotiation.cc:407] Connection header received
0928 02:14:12.111686 (+ 758us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:14:12.111697 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:14:12.111980 (+ 283us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:14:12.112192 (+ 212us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:14:12.112750 (+ 558us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:14:12.114457 (+ 1707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:14:12.116981 (+ 2524us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:14:12.118370 (+ 1389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:14:12.118514 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:14:12.118762 (+ 248us) server_negotiation.cc:1035] Waiting for connection context
0928 02:14:12.120279 (+ 1517us) server_negotiation.cc:299] Negotiation successful
0928 02:14:12.120435 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":660,"thread_start_us":528,"threads_started":1}
I20260928 02:14:12.121258 21462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:14:12.111083 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.235.65:37261 (local address 127.0.0.1:47536)
0928 02:14:12.111719 (+ 636us) negotiation.cc:107] Waiting for socket to connect
0928 02:14:12.111738 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0928 02:14:12.111845 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:14:12.113410 (+ 1565us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:14:12.113415 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:14:12.113615 (+ 200us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:14:12.113970 (+ 355us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:14:12.113982 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:14:12.116358 (+ 2376us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:14:12.116364 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:14:12.117959 (+ 1595us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:14:12.117975 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:14:12.120048 (+ 2073us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:14:12.120062 (+ 14us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:14:12.120593 (+ 531us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:14:12.120633 (+ 40us) client_negotiation.cc:769] Sending connection context
0928 02:14:12.120730 (+ 97us) client_negotiation.cc:240] Negotiation successful
0928 02:14:12.120877 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":470,"thread_start_us":497,"threads_started":1}
I20260928 02:14:12.121994 21463 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:14:12.111426 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47536 (local address 127.6.235.65:37261)
0928 02:14:12.112530 (+ 1104us) server_negotiation.cc:206] Beginning negotiation
0928 02:14:12.112538 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 02:14:12.112563 (+ 25us) server_negotiation.cc:407] Connection header received
0928 02:14:12.112651 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:14:12.112685 (+ 34us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:14:12.112958 (+ 273us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:14:12.113151 (+ 193us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:14:12.114377 (+ 1226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:14:12.116200 (+ 1823us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:14:12.118151 (+ 1951us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:14:12.119536 (+ 1385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:14:12.119671 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:14:12.119894 (+ 223us) server_negotiation.cc:1035] Waiting for connection context
0928 02:14:12.121471 (+ 1577us) server_negotiation.cc:299] Negotiation successful
0928 02:14:12.121635 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":919,"thread_start_us":980,"threads_started":1}
I20260928 02:14:12.124532 21246 tablet_service.cc:1467] Processing CreateTablet for tablet e25fc0f75abc4691a00ada1d06f1edf7 (DEFAULT_TABLE table=test-table [id=29f119d12ea449db9521081aad510dec]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:14:12.124529 21380 tablet_service.cc:1467] Processing CreateTablet for tablet e25fc0f75abc4691a00ada1d06f1edf7 (DEFAULT_TABLE table=test-table [id=29f119d12ea449db9521081aad510dec]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:14:12.125743 21380 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e25fc0f75abc4691a00ada1d06f1edf7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:14:12.125746 21246 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e25fc0f75abc4691a00ada1d06f1edf7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:14:12.130781 21112 tablet_service.cc:1467] Processing CreateTablet for tablet e25fc0f75abc4691a00ada1d06f1edf7 (DEFAULT_TABLE table=test-table [id=29f119d12ea449db9521081aad510dec]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:14:12.132459 21112 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e25fc0f75abc4691a00ada1d06f1edf7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:14:12.139962 21464 tablet_bootstrap.cc:492] T e25fc0f75abc4691a00ada1d06f1edf7 P f820529e7c8e4f2f8341f373b1def2d0: Bootstrap starting.
I20260928 02:14:12.143393 21464 tablet_bootstrap.cc:654] T e25fc0f75abc4691a00ada1d06f1edf7 P f820529e7c8e4f2f8341f373b1def2d0: Neither blocks nor log segments found. Creating new log.
I20260928 02:14:12.144683 21464 log.cc:824] T e25fc0f75abc4691a00ada1d06f1edf7 P f820529e7c8e4f2f8341f373b1def2d0: Log is configured to *not* fsync() on all Append() calls
I20260928 02:14:12.146867 21465 tablet_bootstrap.cc:492] T e25fc0f75abc4691a00ada1d06f1edf7 P f11b50d0630c4d91be9b17973fcb0230: Bootstrap starting.
I20260928 02:14:12.149151 21467 tablet_bootstrap.cc:492] T e25fc0f75abc4691a00ada1d06f1edf7 P 37070660923a4173b9bfe2c5b3abc0c4: Bootstrap starting.
I20260928 02:14:12.150578 21464 tablet_bootstrap.cc:492] T e25fc0f75abc4691a00ada1d06f1edf7 P f820529e7c8e4f2f8341f373b1def2d0: No bootstrap required, opened a new log
I20260928 02:14:12.150820 21465 tablet_bootstrap.cc:654] T e25fc0f75abc4691a00ada1d06f1edf7 P f11b50d0630c4d91be9b17973fcb0230: Neither blocks nor log segments found. Creating new log.
I20260928 02:14:12.150894 21464 ts_tablet_manager.cc:1397] T e25fc0f75abc4691a00ada1d06f1edf7 P f820529e7c8e4f2f8341f373b1def2d0: Time spent bootstrapping tablet: real 0.011s user 0.004s sys 0.004s
I20260928 02:14:12.152176 21467 tablet_bootstrap.cc:654] T e25fc0f75abc4691a00ada1d06f1edf7 P 37070660923a4173b9bfe2c5b3abc0c4: Neither blocks nor log segments found. Creating new log.
I20260928 02:14:12.152184 21465 log.cc:824] T e25fc0f75abc4691a00ada1d06f1edf7 P f11b50d0630c4d91be9b17973fcb0230: Log is configured to *not* fsync() on all Append() calls
I20260928 02:14:12.153400 21467 log.cc:824] T e25fc0f75abc4691a00ada1d06f1edf7 P 37070660923a4173b9bfe2c5b3abc0c4: Log is configured to *not* fsync() on all Append() calls
I20260928 02:14:12.155575 21465 tablet_bootstrap.cc:492] T e25fc0f75abc4691a00ada1d06f1edf7 P f11b50d0630c4d91be9b17973fcb0230: No bootstrap required, opened a new log
I20260928 02:14:12.155869 21465 ts_tablet_manager.cc:1397] T e25fc0f75abc4691a00ada1d06f1edf7 P f11b50d0630c4d91be9b17973fcb0230: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20260928 02:14:12.157404 21467 tablet_bootstrap.cc:492] T e25fc0f75abc4691a00ada1d06f1edf7 P 37070660923a4173b9bfe2c5b3abc0c4: No bootstrap required, opened a new log
I20260928 02:14:12.157678 21467 ts_tablet_manager.cc:1397] T e25fc0f75abc4691a00ada1d06f1edf7 P 37070660923a4173b9bfe2c5b3abc0c4: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.001s
I20260928 02:14:12.159601 21464 raft_consensus.cc:348] T e25fc0f75abc4691a00ada1d06f1edf7 P f820529e7c8e4f2f8341f373b1def2d0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f820529e7c8e4f2f8341f373b1def2d0" member_type: VOTER last_known_addr { host: "127.6.235.66" port: 36697 } } peers { permanent_uuid: "f11b50d0630c4d91be9b17973fcb0230" member_type: VOTER last_known_addr { host: "127.6.235.67" port: 41399 } } peers { permanent_uuid: "37070660923a4173b9bfe2c5b3abc0c4" member_type: VOTER last_known_addr { host: "127.6.235.65" port: 37261 } }
I20260928 02:14:12.159981 21464 raft_consensus.cc:374] T e25fc0f75abc4691a00ada1d06f1edf7 P f820529e7c8e4f2f8341f373b1def2d0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:14:12.160079 21464 raft_consensus.cc:729] T e25fc0f75abc4691a00ada1d06f1edf7 P f820529e7c8e4f2f8341f373b1def2d0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f820529e7c8e4f2f8341f373b1def2d0, State: Initialized, Role: FOLLOWER
I20260928 02:14:12.160691 21464 consensus_queue.cc:260] T e25fc0f75abc4691a00ada1d06f1edf7 P f820529e7c8e4f2f8341f373b1def2d0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f820529e7c8e4f2f8341f373b1def2d0" member_type: VOTER last_known_addr { host: "127.6.235.66" port: 36697 } } peers { permanent_uuid: "f11b50d0630c4d91be9b17973fcb0230" member_type: VOTER last_known_addr { host: "127.6.235.67" port: 41399 } } peers { permanent_uuid: "37070660923a4173b9bfe2c5b3abc0c4" member_type: VOTER last_known_addr { host: "127.6.235.65" port: 37261 } }
I20260928 02:14:12.161844 21311 heartbeater.cc:503] Master 127.6.235.126:40053 was elected leader, sending a full tablet report...
I20260928 02:14:12.162523 21464 ts_tablet_manager.cc:1428] T e25fc0f75abc4691a00ada1d06f1edf7 P f820529e7c8e4f2f8341f373b1def2d0: Time spent starting tablet: real 0.012s user 0.002s sys 0.010s
I20260928 02:14:12.166848 21465 raft_consensus.cc:348] T e25fc0f75abc4691a00ada1d06f1edf7 P f11b50d0630c4d91be9b17973fcb0230 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f820529e7c8e4f2f8341f373b1def2d0" member_type: VOTER last_known_addr { host: "127.6.235.66" port: 36697 } } peers { permanent_uuid: "f11b50d0630c4d91be9b17973fcb0230" member_type: VOTER last_known_addr { host: "127.6.235.67" port: 41399 } } peers { permanent_uuid: "37070660923a4173b9bfe2c5b3abc0c4" member_type: VOTER last_known_addr { host: "127.6.235.65" port: 37261 } }
I20260928 02:14:12.167333 21465 raft_consensus.cc:374] T e25fc0f75abc4691a00ada1d06f1edf7 P f11b50d0630c4d91be9b17973fcb0230 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:14:12.167430 21465 raft_consensus.cc:729] T e25fc0f75abc4691a00ada1d06f1edf7 P f11b50d0630c4d91be9b17973fcb0230 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f11b50d0630c4d91be9b17973fcb0230, State: Initialized, Role: FOLLOWER
I20260928 02:14:12.168077 21465 consensus_queue.cc:260] T e25fc0f75abc4691a00ada1d06f1edf7 P f11b50d0630c4d91be9b17973fcb0230 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f820529e7c8e4f2f8341f373b1def2d0" member_type: VOTER last_known_addr { host: "127.6.235.66" port: 36697 } } peers { permanent_uuid: "f11b50d0630c4d91be9b17973fcb0230" member_type: VOTER last_known_addr { host: "127.6.235.67" port: 41399 } } peers { permanent_uuid: "37070660923a4173b9bfe2c5b3abc0c4" member_type: VOTER last_known_addr { host: "127.6.235.65" port: 37261 } }
I20260928 02:14:12.168268 21467 raft_consensus.cc:348] T e25fc0f75abc4691a00ada1d06f1edf7 P 37070660923a4173b9bfe2c5b3abc0c4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f820529e7c8e4f2f8341f373b1def2d0" member_type: VOTER last_known_addr { host: "127.6.235.66" port: 36697 } } peers { permanent_uuid: "f11b50d0630c4d91be9b17973fcb0230" member_type: VOTER last_known_addr { host: "127.6.235.67" port: 41399 } } peers { permanent_uuid: "37070660923a4173b9bfe2c5b3abc0c4" member_type: VOTER last_known_addr { host: "127.6.235.65" port: 37261 } }
I20260928 02:14:12.168787 21467 raft_consensus.cc:374] T e25fc0f75abc4691a00ada1d06f1edf7 P 37070660923a4173b9bfe2c5b3abc0c4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:14:12.168908 21467 raft_consensus.cc:729] T e25fc0f75abc4691a00ada1d06f1edf7 P 37070660923a4173b9bfe2c5b3abc0c4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 37070660923a4173b9bfe2c5b3abc0c4, State: Initialized, Role: FOLLOWER
I20260928 02:14:12.169677 21445 heartbeater.cc:503] Master 127.6.235.126:40053 was elected leader, sending a full tablet report...
I20260928 02:14:12.169613 21467 consensus_queue.cc:260] T e25fc0f75abc4691a00ada1d06f1edf7 P 37070660923a4173b9bfe2c5b3abc0c4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f820529e7c8e4f2f8341f373b1def2d0" member_type: VOTER last_known_addr { host: "127.6.235.66" port: 36697 } } peers { permanent_uuid: "f11b50d0630c4d91be9b17973fcb0230" member_type: VOTER last_known_addr { host: "127.6.235.67" port: 41399 } } peers { permanent_uuid: "37070660923a4173b9bfe2c5b3abc0c4" member_type: VOTER last_known_addr { host: "127.6.235.65" port: 37261 } }
I20260928 02:14:12.170136 21465 ts_tablet_manager.cc:1428] T e25fc0f75abc4691a00ada1d06f1edf7 P f11b50d0630c4d91be9b17973fcb0230: Time spent starting tablet: real 0.014s user 0.015s sys 0.000s
I20260928 02:14:12.171314 21467 ts_tablet_manager.cc:1428] T e25fc0f75abc4691a00ada1d06f1edf7 P 37070660923a4173b9bfe2c5b3abc0c4: Time spent starting tablet: real 0.014s user 0.008s sys 0.005s
W20260928 02:14:12.198850 21178 tablet.cc:2369] T e25fc0f75abc4691a00ada1d06f1edf7 P 37070660923a4173b9bfe2c5b3abc0c4: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 02:14:12.238657 21312 tablet.cc:2369] T e25fc0f75abc4691a00ada1d06f1edf7 P f820529e7c8e4f2f8341f373b1def2d0: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 02:14:12.286720 21446 tablet.cc:2369] T e25fc0f75abc4691a00ada1d06f1edf7 P f11b50d0630c4d91be9b17973fcb0230: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 02:14:12.443711 21470 raft_consensus.cc:482] T e25fc0f75abc4691a00ada1d06f1edf7 P f820529e7c8e4f2f8341f373b1def2d0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 02:14:12.444039 21470 raft_consensus.cc:504] T e25fc0f75abc4691a00ada1d06f1edf7 P f820529e7c8e4f2f8341f373b1def2d0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f820529e7c8e4f2f8341f373b1def2d0" member_type: VOTER last_known_addr { host: "127.6.235.66" port: 36697 } } peers { permanent_uuid: "f11b50d0630c4d91be9b17973fcb0230" member_type: VOTER last_known_addr { host: "127.6.235.67" port: 41399 } } peers { permanent_uuid: "37070660923a4173b9bfe2c5b3abc0c4" member_type: VOTER last_known_addr { host: "127.6.235.65" port: 37261 } }
I20260928 02:14:12.445734 21470 leader_election.cc:288] T e25fc0f75abc4691a00ada1d06f1edf7 P f820529e7c8e4f2f8341f373b1def2d0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f11b50d0630c4d91be9b17973fcb0230 (127.6.235.67:41399), 37070660923a4173b9bfe2c5b3abc0c4 (127.6.235.65:37261)
I20260928 02:14:12.452968 21460 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:14:12.446155 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.235.66:37919 (local address 127.6.235.67:41399)
0928 02:14:12.446413 (+ 258us) server_negotiation.cc:206] Beginning negotiation
0928 02:14:12.446453 (+ 40us) server_negotiation.cc:399] Waiting for connection header
0928 02:14:12.446834 (+ 381us) server_negotiation.cc:407] Connection header received
0928 02:14:12.447051 (+ 217us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:14:12.447059 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:14:12.447154 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:14:12.447252 (+ 98us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:14:12.447948 (+ 696us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:14:12.448949 (+ 1001us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:14:12.450928 (+ 1979us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:14:12.451924 (+ 996us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:14:12.452048 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:14:12.452101 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0928 02:14:12.452543 (+ 442us) server_negotiation.cc:299] Negotiation successful
0928 02:14:12.452678 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260928 02:14:12.453078 21473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:14:12.446109 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.235.67:41399 (local address 127.6.235.66:37919)
0928 02:14:12.446667 (+ 558us) negotiation.cc:107] Waiting for socket to connect
0928 02:14:12.446683 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 02:14:12.446878 (+ 195us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:14:12.447350 (+ 472us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:14:12.447354 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:14:12.447380 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:14:12.447787 (+ 407us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:14:12.447801 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:14:12.449082 (+ 1281us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:14:12.449088 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:14:12.450765 (+ 1677us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:14:12.450778 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:14:12.452151 (+ 1373us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:14:12.452157 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:14:12.452335 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:14:12.452377 (+ 42us) client_negotiation.cc:769] Sending connection context
0928 02:14:12.452494 (+ 117us) client_negotiation.cc:240] Negotiation successful
0928 02:14:12.452636 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":451,"thread_start_us":308,"threads_started":1}
I20260928 02:14:12.453603 21463 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:14:12.447026 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.235.66:56381 (local address 127.6.235.65:37261)
0928 02:14:12.447247 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0928 02:14:12.447252 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 02:14:12.447598 (+ 346us) server_negotiation.cc:407] Connection header received
0928 02:14:12.447721 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:14:12.447728 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:14:12.447808 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:14:12.447906 (+ 98us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:14:12.448521 (+ 615us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:14:12.449458 (+ 937us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:14:12.451477 (+ 2019us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:14:12.452559 (+ 1082us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:14:12.452702 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:14:12.452783 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0928 02:14:12.453216 (+ 433us) server_negotiation.cc:299] Negotiation successful
0928 02:14:12.453333 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260928 02:14:12.454102 21474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:14:12.446833 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.235.65:37261 (local address 127.6.235.66:56381)
0928 02:14:12.447485 (+ 652us) negotiation.cc:107] Waiting for socket to connect
0928 02:14:12.447499 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 02:14:12.447601 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:14:12.447963 (+ 362us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:14:12.447967 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:14:12.447993 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:14:12.448350 (+ 357us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:14:12.448361 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:14:12.449592 (+ 1231us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:14:12.449596 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:14:12.451312 (+ 1716us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:14:12.451327 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:14:12.452779 (+ 1452us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:14:12.452786 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:14:12.453015 (+ 229us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:14:12.453051 (+ 36us) client_negotiation.cc:769] Sending connection context
0928 02:14:12.453714 (+ 663us) client_negotiation.cc:240] Negotiation successful
0928 02:14:12.453869 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":525,"thread_start_us":305,"threads_started":1}
I20260928 02:14:12.454257 21400 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e25fc0f75abc4691a00ada1d06f1edf7" candidate_uuid: "f820529e7c8e4f2f8341f373b1def2d0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f11b50d0630c4d91be9b17973fcb0230" is_pre_election: true
I20260928 02:14:12.454869 21400 raft_consensus.cc:2454] T e25fc0f75abc4691a00ada1d06f1edf7 P f11b50d0630c4d91be9b17973fcb0230 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f820529e7c8e4f2f8341f373b1def2d0 in term 0.
I20260928 02:14:12.455152 21132 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e25fc0f75abc4691a00ada1d06f1edf7" candidate_uuid: "f820529e7c8e4f2f8341f373b1def2d0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "37070660923a4173b9bfe2c5b3abc0c4" is_pre_election: true
I20260928 02:14:12.455582 21199 leader_election.cc:302] T e25fc0f75abc4691a00ada1d06f1edf7 P f820529e7c8e4f2f8341f373b1def2d0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: f11b50d0630c4d91be9b17973fcb0230, f820529e7c8e4f2f8341f373b1def2d0; no voters:
I20260928 02:14:12.455699 21132 raft_consensus.cc:2454] T e25fc0f75abc4691a00ada1d06f1edf7 P 37070660923a4173b9bfe2c5b3abc0c4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f820529e7c8e4f2f8341f373b1def2d0 in term 0.
I20260928 02:14:12.456008 21470 raft_consensus.cc:2781] T e25fc0f75abc4691a00ada1d06f1edf7 P f820529e7c8e4f2f8341f373b1def2d0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 02:14:12.456169 21470 raft_consensus.cc:482] T e25fc0f75abc4691a00ada1d06f1edf7 P f820529e7c8e4f2f8341f373b1def2d0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 02:14:12.456305 21470 raft_consensus.cc:3037] T e25fc0f75abc4691a00ada1d06f1edf7 P f820529e7c8e4f2f8341f373b1def2d0 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:14:12.458669 21470 raft_consensus.cc:504] T e25fc0f75abc4691a00ada1d06f1edf7 P f820529e7c8e4f2f8341f373b1def2d0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f820529e7c8e4f2f8341f373b1def2d0" member_type: VOTER last_known_addr { host: "127.6.235.66" port: 36697 } } peers { permanent_uuid: "f11b50d0630c4d91be9b17973fcb0230" member_type: VOTER last_known_addr { host: "127.6.235.67" port: 41399 } } peers { permanent_uuid: "37070660923a4173b9bfe2c5b3abc0c4" member_type: VOTER last_known_addr { host: "127.6.235.65" port: 37261 } }
I20260928 02:14:12.459270 21470 leader_election.cc:288] T e25fc0f75abc4691a00ada1d06f1edf7 P f820529e7c8e4f2f8341f373b1def2d0 [CANDIDATE]: Term 1 election: Requested vote from peers f11b50d0630c4d91be9b17973fcb0230 (127.6.235.67:41399), 37070660923a4173b9bfe2c5b3abc0c4 (127.6.235.65:37261)
I20260928 02:14:12.459798 21400 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e25fc0f75abc4691a00ada1d06f1edf7" candidate_uuid: "f820529e7c8e4f2f8341f373b1def2d0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f11b50d0630c4d91be9b17973fcb0230"
I20260928 02:14:12.459903 21132 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e25fc0f75abc4691a00ada1d06f1edf7" candidate_uuid: "f820529e7c8e4f2f8341f373b1def2d0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "37070660923a4173b9bfe2c5b3abc0c4"
I20260928 02:14:12.460038 21400 raft_consensus.cc:3037] T e25fc0f75abc4691a00ada1d06f1edf7 P f11b50d0630c4d91be9b17973fcb0230 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:14:12.460155 21132 raft_consensus.cc:3037] T e25fc0f75abc4691a00ada1d06f1edf7 P 37070660923a4173b9bfe2c5b3abc0c4 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:14:12.462404 21400 raft_consensus.cc:2454] T e25fc0f75abc4691a00ada1d06f1edf7 P f11b50d0630c4d91be9b17973fcb0230 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f820529e7c8e4f2f8341f373b1def2d0 in term 1.
I20260928 02:14:12.462435 21132 raft_consensus.cc:2454] T e25fc0f75abc4691a00ada1d06f1edf7 P 37070660923a4173b9bfe2c5b3abc0c4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f820529e7c8e4f2f8341f373b1def2d0 in term 1.
I20260928 02:14:12.462919 21199 leader_election.cc:302] T e25fc0f75abc4691a00ada1d06f1edf7 P f820529e7c8e4f2f8341f373b1def2d0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: f11b50d0630c4d91be9b17973fcb0230, f820529e7c8e4f2f8341f373b1def2d0; no voters:
I20260928 02:14:12.463212 21470 raft_consensus.cc:2781] T e25fc0f75abc4691a00ada1d06f1edf7 P f820529e7c8e4f2f8341f373b1def2d0 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:14:12.463670 21470 raft_consensus.cc:686] T e25fc0f75abc4691a00ada1d06f1edf7 P f820529e7c8e4f2f8341f373b1def2d0 [term 1 LEADER]: Becoming Leader. State: Replica: f820529e7c8e4f2f8341f373b1def2d0, State: Running, Role: LEADER
I20260928 02:14:12.464156 21470 consensus_queue.cc:237] T e25fc0f75abc4691a00ada1d06f1edf7 P f820529e7c8e4f2f8341f373b1def2d0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f820529e7c8e4f2f8341f373b1def2d0" member_type: VOTER last_known_addr { host: "127.6.235.66" port: 36697 } } peers { permanent_uuid: "f11b50d0630c4d91be9b17973fcb0230" member_type: VOTER last_known_addr { host: "127.6.235.67" port: 41399 } } peers { permanent_uuid: "37070660923a4173b9bfe2c5b3abc0c4" member_type: VOTER last_known_addr { host: "127.6.235.65" port: 37261 } }
I20260928 02:14:12.469152 20988 catalog_manager.cc:5495] T e25fc0f75abc4691a00ada1d06f1edf7 P f820529e7c8e4f2f8341f373b1def2d0 reported cstate change: term changed from 0 to 1, leader changed from <none> to f820529e7c8e4f2f8341f373b1def2d0 (127.6.235.66). New cstate: current_term: 1 leader_uuid: "f820529e7c8e4f2f8341f373b1def2d0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f820529e7c8e4f2f8341f373b1def2d0" member_type: VOTER last_known_addr { host: "127.6.235.66" port: 36697 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f11b50d0630c4d91be9b17973fcb0230" member_type: VOTER last_known_addr { host: "127.6.235.67" port: 41399 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "37070660923a4173b9bfe2c5b3abc0c4" member_type: VOTER last_known_addr { host: "127.6.235.65" port: 37261 } health_report { overall_health: UNKNOWN } } }
I20260928 02:14:12.494215 21459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:14:12.487672 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48312 (local address 127.6.235.66:36697)
0928 02:14:12.487895 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0928 02:14:12.487899 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:14:12.487927 (+ 28us) server_negotiation.cc:407] Connection header received
0928 02:14:12.488182 (+ 255us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:14:12.488186 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:14:12.488269 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:14:12.488385 (+ 116us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 02:14:12.488816 (+ 431us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:14:12.489699 (+ 883us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:14:12.491716 (+ 2017us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:14:12.492157 (+ 441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:14:12.492252 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:14:12.493012 (+ 760us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 02:14:12.493588 (+ 576us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 02:14:12.493702 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0928 02:14:12.493831 (+ 129us) server_negotiation.cc:299] Negotiation successful
0928 02:14:12.493943 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87}
I20260928 02:14:12.502641 21132 raft_consensus.cc:1261] T e25fc0f75abc4691a00ada1d06f1edf7 P 37070660923a4173b9bfe2c5b3abc0c4 [term 1 FOLLOWER]: Refusing update from remote peer f820529e7c8e4f2f8341f373b1def2d0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 02:14:12.502770 21400 raft_consensus.cc:1261] T e25fc0f75abc4691a00ada1d06f1edf7 P f11b50d0630c4d91be9b17973fcb0230 [term 1 FOLLOWER]: Refusing update from remote peer f820529e7c8e4f2f8341f373b1def2d0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 02:14:12.503690 21475 consensus_queue.cc:1035] T e25fc0f75abc4691a00ada1d06f1edf7 P f820529e7c8e4f2f8341f373b1def2d0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "37070660923a4173b9bfe2c5b3abc0c4" member_type: VOTER last_known_addr { host: "127.6.235.65" port: 37261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 02:14:12.504092 21470 consensus_queue.cc:1035] T e25fc0f75abc4691a00ada1d06f1edf7 P f820529e7c8e4f2f8341f373b1def2d0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f11b50d0630c4d91be9b17973fcb0230" member_type: VOTER last_known_addr { host: "127.6.235.67" port: 41399 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 02:14:12.517271 21479 mvcc.cc:205] Tried to move back new op lower bound from 7334140528640888832 to 7334140528497737728. Current Snapshot: MvccSnapshot[applied={T|T < 7334140528640888832}]
I20260928 02:14:12.530387 20987 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35006:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 02:14:12.530757 20987 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35006:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 02:14:12.535087 20987 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 65164f35d42e4ab5a76865f2884ee008: Sending DeleteTablet for 3 replicas of tablet e25fc0f75abc4691a00ada1d06f1edf7
I20260928 02:14:12.536456 21380 tablet_service.cc:1514] Processing DeleteTablet for tablet e25fc0f75abc4691a00ada1d06f1edf7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:14:12 UTC) from {username='kudu', principal='kudu/127.6.235.126@KRBTEST.COM'} at 127.0.0.1:58842
I20260928 02:14:12.536437 21246 tablet_service.cc:1514] Processing DeleteTablet for tablet e25fc0f75abc4691a00ada1d06f1edf7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:14:12 UTC) from {username='kudu', principal='kudu/127.6.235.126@KRBTEST.COM'} at 127.0.0.1:48296
I20260928 02:14:12.536661 7085 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskTlaUuc/build/asan/bin/kudu pid 21049
I20260928 02:14:12.537547 21112 tablet_service.cc:1514] Processing DeleteTablet for tablet e25fc0f75abc4691a00ada1d06f1edf7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:14:12 UTC) from {username='kudu', principal='kudu/127.6.235.126@KRBTEST.COM'} at 127.0.0.1:47536
I20260928 02:14:12.537693 21488 tablet_replica.cc:330] T e25fc0f75abc4691a00ada1d06f1edf7 P f820529e7c8e4f2f8341f373b1def2d0: stopping tablet replica
I20260928 02:14:12.538159 21488 raft_consensus.cc:2229] T e25fc0f75abc4691a00ada1d06f1edf7 P f820529e7c8e4f2f8341f373b1def2d0 [term 1 LEADER]: Raft consensus shutting down.
I20260928 02:14:12.538975 21488 raft_consensus.cc:2258] T e25fc0f75abc4691a00ada1d06f1edf7 P f820529e7c8e4f2f8341f373b1def2d0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 02:14:12.541733 21488 ts_tablet_manager.cc:1905] T e25fc0f75abc4691a00ada1d06f1edf7 P f820529e7c8e4f2f8341f373b1def2d0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 02:14:12.546066 21489 tablet_replica.cc:330] T e25fc0f75abc4691a00ada1d06f1edf7 P f11b50d0630c4d91be9b17973fcb0230: stopping tablet replica
I20260928 02:14:12.546659 21489 raft_consensus.cc:2229] T e25fc0f75abc4691a00ada1d06f1edf7 P f11b50d0630c4d91be9b17973fcb0230 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 02:14:12.547317 21489 raft_consensus.cc:2258] T e25fc0f75abc4691a00ada1d06f1edf7 P f11b50d0630c4d91be9b17973fcb0230 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 02:14:12.549726 21488 ts_tablet_manager.cc:1918] T e25fc0f75abc4691a00ada1d06f1edf7 P f820529e7c8e4f2f8341f373b1def2d0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 02:14:12.549926 21488 log.cc:1196] T e25fc0f75abc4691a00ada1d06f1edf7 P f820529e7c8e4f2f8341f373b1def2d0: Deleting WAL directory at /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790561432251112-7085-0/minicluster-data/ts-1/wal/wals/e25fc0f75abc4691a00ada1d06f1edf7
I20260928 02:14:12.550505 21488 ts_tablet_manager.cc:1939] T e25fc0f75abc4691a00ada1d06f1edf7 P f820529e7c8e4f2f8341f373b1def2d0: Deleting consensus metadata
I20260928 02:14:12.552203 21491 tablet_replica.cc:330] T e25fc0f75abc4691a00ada1d06f1edf7 P 37070660923a4173b9bfe2c5b3abc0c4: stopping tablet replica
I20260928 02:14:12.552520 21489 ts_tablet_manager.cc:1905] T e25fc0f75abc4691a00ada1d06f1edf7 P f11b50d0630c4d91be9b17973fcb0230: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 02:14:12.552464 20973 catalog_manager.cc:4841] TS f820529e7c8e4f2f8341f373b1def2d0 (127.6.235.66:36697): tablet e25fc0f75abc4691a00ada1d06f1edf7 (table test-table [id=29f119d12ea449db9521081aad510dec]) successfully deleted
I20260928 02:14:12.552883 21491 raft_consensus.cc:2229] T e25fc0f75abc4691a00ada1d06f1edf7 P 37070660923a4173b9bfe2c5b3abc0c4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 02:14:12.553576 21491 raft_consensus.cc:2258] T e25fc0f75abc4691a00ada1d06f1edf7 P 37070660923a4173b9bfe2c5b3abc0c4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 02:14:12.555838 21491 ts_tablet_manager.cc:1905] T e25fc0f75abc4691a00ada1d06f1edf7 P 37070660923a4173b9bfe2c5b3abc0c4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 02:14:12.559988 21489 ts_tablet_manager.cc:1918] T e25fc0f75abc4691a00ada1d06f1edf7 P f11b50d0630c4d91be9b17973fcb0230: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 02:14:12.560171 21489 log.cc:1196] T e25fc0f75abc4691a00ada1d06f1edf7 P f11b50d0630c4d91be9b17973fcb0230: Deleting WAL directory at /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790561432251112-7085-0/minicluster-data/ts-2/wal/wals/e25fc0f75abc4691a00ada1d06f1edf7
I20260928 02:14:12.560705 21489 ts_tablet_manager.cc:1939] T e25fc0f75abc4691a00ada1d06f1edf7 P f11b50d0630c4d91be9b17973fcb0230: Deleting consensus metadata
Sep 28 02:14:12 dist-test-slave-86gv krb5kdc[20943](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561649, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.235.65@KRBTEST.COM
I20260928 02:14:12.562636 20972 catalog_manager.cc:4841] TS f11b50d0630c4d91be9b17973fcb0230 (127.6.235.67:41399): tablet e25fc0f75abc4691a00ada1d06f1edf7 (table test-table [id=29f119d12ea449db9521081aad510dec]) successfully deleted
I20260928 02:14:12.563308 21491 ts_tablet_manager.cc:1918] T e25fc0f75abc4691a00ada1d06f1edf7 P 37070660923a4173b9bfe2c5b3abc0c4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 02:14:12.563510 21491 log.cc:1196] T e25fc0f75abc4691a00ada1d06f1edf7 P 37070660923a4173b9bfe2c5b3abc0c4: Deleting WAL directory at /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790561432251112-7085-0/minicluster-data/ts-0/wal/wals/e25fc0f75abc4691a00ada1d06f1edf7
I20260928 02:14:12.564034 21491 ts_tablet_manager.cc:1939] T e25fc0f75abc4691a00ada1d06f1edf7 P 37070660923a4173b9bfe2c5b3abc0c4: Deleting consensus metadata
I20260928 02:14:12.565719 20972 catalog_manager.cc:4841] TS 37070660923a4173b9bfe2c5b3abc0c4 (127.6.235.65:37261): tablet e25fc0f75abc4691a00ada1d06f1edf7 (table test-table [id=29f119d12ea449db9521081aad510dec]) successfully deleted
I20260928 02:14:12.570292 21463 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:14:12.538559 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47546 (local address 127.6.235.65:37261)
0928 02:14:12.538779 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0928 02:14:12.538789 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 02:14:12.545084 (+ 6295us) server_negotiation.cc:407] Connection header received
0928 02:14:12.545304 (+ 220us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:14:12.545309 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:14:12.545402 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:14:12.545556 (+ 154us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:14:12.553015 (+ 7459us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:14:12.554264 (+ 1249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:14:12.556476 (+ 2212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:14:12.557209 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:14:12.557319 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:14:12.563044 (+ 5725us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:14:12.563060 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:14:12.563078 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:14:12.563131 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:14:12.567156 (+ 4025us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:14:12.567888 (+ 732us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:14:12.567897 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:14:12.567900 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:14:12.568073 (+ 173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:14:12.568466 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:14:12.568471 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:14:12.568475 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:14:12.568841 (+ 366us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:14:12.569278 (+ 437us) server_negotiation.cc:1035] Waiting for connection context
0928 02:14:12.569657 (+ 379us) server_negotiation.cc:299] Negotiation successful
0928 02:14:12.569960 (+ 303us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260928 02:14:12.571034 21172 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 02:14:12.667640 7085 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTlaUuc/build/asan/bin/kudu with pid 21049
I20260928 02:14:12.686587 7085 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskTlaUuc/build/asan/bin/kudu pid 21183
Sep 28 02:14:12 dist-test-slave-86gv krb5kdc[20943](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561649, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.235.66@KRBTEST.COM
I20260928 02:14:12.704428 21459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:14:12.687495 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48322 (local address 127.6.235.66:36697)
0928 02:14:12.687693 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0928 02:14:12.687700 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 02:14:12.687721 (+ 21us) server_negotiation.cc:407] Connection header received
0928 02:14:12.687788 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:14:12.687793 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:14:12.687904 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:14:12.688018 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:14:12.689369 (+ 1351us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:14:12.690597 (+ 1228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:14:12.691875 (+ 1278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:14:12.692611 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:14:12.692739 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:14:12.697389 (+ 4650us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:14:12.697417 (+ 28us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:14:12.697439 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:14:12.697517 (+ 78us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:14:12.701536 (+ 4019us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:14:12.702262 (+ 726us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:14:12.702271 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:14:12.702276 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:14:12.702461 (+ 185us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:14:12.702805 (+ 344us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:14:12.702811 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:14:12.702814 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:14:12.703276 (+ 462us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:14:12.703603 (+ 327us) server_negotiation.cc:1035] Waiting for connection context
0928 02:14:12.703789 (+ 186us) server_negotiation.cc:299] Negotiation successful
0928 02:14:12.704019 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260928 02:14:12.705332 21306 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 02:14:12.820780 7085 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTlaUuc/build/asan/bin/kudu with pid 21183
I20260928 02:14:12.840041 7085 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskTlaUuc/build/asan/bin/kudu pid 21317
Sep 28 02:14:12 dist-test-slave-86gv krb5kdc[20943](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561649, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.235.67@KRBTEST.COM
I20260928 02:14:12.856037 21460 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:14:12.840822 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58858 (local address 127.6.235.67:41399)
0928 02:14:12.841047 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0928 02:14:12.841054 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 02:14:12.841150 (+ 96us) server_negotiation.cc:407] Connection header received
0928 02:14:12.841403 (+ 253us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:14:12.841408 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:14:12.841486 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:14:12.841602 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:14:12.842881 (+ 1279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:14:12.843616 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:14:12.845017 (+ 1401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:14:12.845432 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:14:12.845572 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:14:12.850459 (+ 4887us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:14:12.850477 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:14:12.850495 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:14:12.850567 (+ 72us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:14:12.853565 (+ 2998us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:14:12.854255 (+ 690us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:14:12.854262 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:14:12.854265 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:14:12.854441 (+ 176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:14:12.854764 (+ 323us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:14:12.854767 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:14:12.854769 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:14:12.855124 (+ 355us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:14:12.855348 (+ 224us) server_negotiation.cc:1035] Waiting for connection context
0928 02:14:12.855616 (+ 268us) server_negotiation.cc:299] Negotiation successful
0928 02:14:12.855784 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
I20260928 02:14:12.856709 21440 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 02:14:12.946928 7085 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTlaUuc/build/asan/bin/kudu with pid 21317
I20260928 02:14:12.964577 7085 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskTlaUuc/build/asan/bin/kudu pid 20956
I20260928 02:14:12.965862 21018 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 02:14:13.077518 7085 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTlaUuc/build/asan/bin/kudu with pid 20956
I20260928 02:14:13.099970 7085 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTlaUuc/build/asan/bin/kudu
/tmp/dist-test-taskTlaUuc/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790561432251112-7085-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790561432251112-7085-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790561432251112-7085-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790561432251112-7085-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.235.126:40053
--webserver_interface=127.6.235.126
--webserver_port=35131
--keytab_file=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790561432251112-7085-0/krb5kdc/kudu_127.6.235.126.keytab
--principal=kudu/127.6.235.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.235.84:34573
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.235.126:40053
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--ipki_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790561432251112-7085-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790561432251112-7085-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790561432251112-7085-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:14:13.439450 21496 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:14:13.439764 21496 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:14:13.439944 21496 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:14:13.447384 21496 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 02:14:13.447513 21496 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:14:13.447623 21496 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:14:13.447710 21496 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 02:14:13.447782 21496 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 02:14:13.458084 21496 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.235.84:34573
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790561432251112-7085-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790561432251112-7085-0/minicluster-data/master-0/wal
--ipki_private_key_password_cmd=echo foo
--ipki_ca_key_size=768
--master_addresses=127.6.235.126:40053
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790561432251112-7085-0/krb5kdc/kudu_127.6.235.126.keytab
--principal=kudu/127.6.235.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.235.126:40053
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790561432251112-7085-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.235.126
--webserver_port=35131
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790561432251112-7085-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 00:51:59 UTC on ccycloud.quasar-axgkwn.root.comops.site
build id 4676464
ASAN enabled
I20260928 02:14:13.459533 21496 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:14:13.460990 21496 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:14:13.485210 21502 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:14:13.485188 21503 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:14:13.485239 21505 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:14:13.485705 21496 server_base.cc:1034] running on GCE node
I20260928 02:14:13.487000 21496 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:14:13.488755 21496 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:14:13.490046 21496 hybrid_clock.cc:648] HybridClock initialized: now 1790561653489979 us; error 73 us; skew 500 ppm
I20260928 02:14:13.490571 21496 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 02:14:13 dist-test-slave-86gv krb5kdc[20943](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561653, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.235.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:14:13.494849 21496 init.cc:377] Logged in from keytab as kudu/127.6.235.126@KRBTEST.COM (short username kudu)
I20260928 02:14:13.496604 21496 webserver.cc:460] Webserver started at http://127.6.235.126:35131/ using document root <none> and password file <none>
I20260928 02:14:13.497479 21496 fs_manager.cc:359] Metadata directory not provided
I20260928 02:14:13.497649 21496 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:14:13.503881 21496 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.002s sys 0.003s
I20260928 02:14:13.507107 21512 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:14:13.508641 21496 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260928 02:14:13.508837 21496 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790561432251112-7085-0/minicluster-data/master-0/data,/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790561432251112-7085-0/minicluster-data/master-0/wal
uuid: "65164f35d42e4ab5a76865f2884ee008"
format_stamp: "Formatted at 2026-09-28 02:14:10 on dist-test-slave-86gv"
server_key: "004236509a5ac5a25bf575733d3d9620"
server_key_iv: "116136b2a19c5a524b3a72651389aeb2"
server_key_version: "encryptionkey@0"
I20260928 02:14:13.510090 21496 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790561432251112-7085-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790561432251112-7085-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790561432251112-7085-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:14:13.548288 21496 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:14:13.553567 21496 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:14:13.553902 21496 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:14:13.573712 21496 rpc_server.cc:307] RPC server started. Bound to: 127.6.235.126:40053
I20260928 02:14:13.573719 21563 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.235.126:40053 every 8 connection(s)
I20260928 02:14:13.575400 21496 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790561432251112-7085-0/minicluster-data/master-0/data/info.pb
I20260928 02:14:13.582856 21564 sys_catalog.cc:263] Verifying existing consensus state
I20260928 02:14:13.584944 7085 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTlaUuc/build/asan/bin/kudu as pid 21496
I20260928 02:14:13.585922 7085 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTlaUuc/build/asan/bin/kudu
/tmp/dist-test-taskTlaUuc/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790561432251112-7085-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790561432251112-7085-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790561432251112-7085-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790561432251112-7085-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.235.65:37261
--local_ip_for_outbound_sockets=127.6.235.65
--tserver_master_addrs=127.6.235.126:40053
--webserver_port=41415
--webserver_interface=127.6.235.65
--keytab_file=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790561432251112-7085-0/krb5kdc/kudu_127.6.235.65.keytab
--principal=kudu/127.6.235.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.235.84:34573
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790561432251112-7085-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790561432251112-7085-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790561432251112-7085-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 02:14:13.587277 21564 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 65164f35d42e4ab5a76865f2884ee008: Bootstrap starting.
I20260928 02:14:13.609864 21564 log.cc:824] T 00000000000000000000000000000000 P 65164f35d42e4ab5a76865f2884ee008: Log is configured to *not* fsync() on all Append() calls
I20260928 02:14:13.622802 21564 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 65164f35d42e4ab5a76865f2884ee008: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 02:14:13.623380 21564 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 65164f35d42e4ab5a76865f2884ee008: Bootstrap complete.
I20260928 02:14:13.632719 21564 raft_consensus.cc:348] T 00000000000000000000000000000000 P 65164f35d42e4ab5a76865f2884ee008 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "65164f35d42e4ab5a76865f2884ee008" member_type: VOTER last_known_addr { host: "127.6.235.126" port: 40053 } }
I20260928 02:14:13.633669 21564 raft_consensus.cc:729] T 00000000000000000000000000000000 P 65164f35d42e4ab5a76865f2884ee008 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 65164f35d42e4ab5a76865f2884ee008, State: Initialized, Role: FOLLOWER
I20260928 02:14:13.634370 21564 consensus_queue.cc:260] T 00000000000000000000000000000000 P 65164f35d42e4ab5a76865f2884ee008 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "65164f35d42e4ab5a76865f2884ee008" member_type: VOTER last_known_addr { host: "127.6.235.126" port: 40053 } }
I20260928 02:14:13.634624 21564 raft_consensus.cc:388] T 00000000000000000000000000000000 P 65164f35d42e4ab5a76865f2884ee008 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 02:14:13.634742 21564 raft_consensus.cc:482] T 00000000000000000000000000000000 P 65164f35d42e4ab5a76865f2884ee008 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 02:14:13.634948 21564 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 65164f35d42e4ab5a76865f2884ee008 [term 1 FOLLOWER]: Advancing to term 2
I20260928 02:14:13.638916 21564 raft_consensus.cc:504] T 00000000000000000000000000000000 P 65164f35d42e4ab5a76865f2884ee008 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "65164f35d42e4ab5a76865f2884ee008" member_type: VOTER last_known_addr { host: "127.6.235.126" port: 40053 } }
I20260928 02:14:13.639544 21564 leader_election.cc:302] T 00000000000000000000000000000000 P 65164f35d42e4ab5a76865f2884ee008 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 65164f35d42e4ab5a76865f2884ee008; no voters:
I20260928 02:14:13.640218 21564 leader_election.cc:288] T 00000000000000000000000000000000 P 65164f35d42e4ab5a76865f2884ee008 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260928 02:14:13.640360 21568 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 65164f35d42e4ab5a76865f2884ee008 [term 2 FOLLOWER]: Leader election won for term 2
I20260928 02:14:13.641870 21568 raft_consensus.cc:686] T 00000000000000000000000000000000 P 65164f35d42e4ab5a76865f2884ee008 [term 2 LEADER]: Becoming Leader. State: Replica: 65164f35d42e4ab5a76865f2884ee008, State: Running, Role: LEADER
I20260928 02:14:13.642480 21568 consensus_queue.cc:237] T 00000000000000000000000000000000 P 65164f35d42e4ab5a76865f2884ee008 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "65164f35d42e4ab5a76865f2884ee008" member_type: VOTER last_known_addr { host: "127.6.235.126" port: 40053 } }
I20260928 02:14:13.642812 21564 sys_catalog.cc:564] T 00000000000000000000000000000000 P 65164f35d42e4ab5a76865f2884ee008 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 02:14:13.645078 21570 sys_catalog.cc:455] T 00000000000000000000000000000000 P 65164f35d42e4ab5a76865f2884ee008 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 65164f35d42e4ab5a76865f2884ee008. Latest consensus state: current_term: 2 leader_uuid: "65164f35d42e4ab5a76865f2884ee008" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "65164f35d42e4ab5a76865f2884ee008" member_type: VOTER last_known_addr { host: "127.6.235.126" port: 40053 } } }
I20260928 02:14:13.645526 21570 sys_catalog.cc:458] T 00000000000000000000000000000000 P 65164f35d42e4ab5a76865f2884ee008 [sys.catalog]: This master's current role is: LEADER
I20260928 02:14:13.646159 21569 sys_catalog.cc:455] T 00000000000000000000000000000000 P 65164f35d42e4ab5a76865f2884ee008 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "65164f35d42e4ab5a76865f2884ee008" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "65164f35d42e4ab5a76865f2884ee008" member_type: VOTER last_known_addr { host: "127.6.235.126" port: 40053 } } }
I20260928 02:14:13.646442 21569 sys_catalog.cc:458] T 00000000000000000000000000000000 P 65164f35d42e4ab5a76865f2884ee008 [sys.catalog]: This master's current role is: LEADER
I20260928 02:14:13.654680 21574 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 02:14:13.665004 21574 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 02:14:13.666951 21574 catalog_manager.cc:1259] Loaded cluster ID: 6dfacd5e0cbd4b6fb1ce99f299c840f2
I20260928 02:14:13.667080 21574 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 02:14:13.671113 21574 catalog_manager.cc:1314] IPKI private key was stored in cleartext, attempting to encrypt it.
I20260928 02:14:13.696312 21574 catalog_manager.cc:1324] IPKI private key successfully encrypted.
I20260928 02:14:13.705592 21574 catalog_manager.cc:1504] Loading token signing keys...
I20260928 02:14:13.707280 21574 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 65164f35d42e4ab5a76865f2884ee008: Loaded TSK: 0
I20260928 02:14:13.708650 21574 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260928 02:14:13.984999 21566 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:14:13.985378 21566 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:14:13.985595 21566 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:14:13.994095 21566 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:14:13.994282 21566 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:14:13.994482 21566 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.235.65
I20260928 02:14:14.006250 21566 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.235.84:34573
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790561432251112-7085-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790561432251112-7085-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790561432251112-7085-0/krb5kdc/kudu_127.6.235.65.keytab
--principal=kudu/127.6.235.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.235.65:37261
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790561432251112-7085-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.6.235.65
--webserver_port=41415
--tserver_master_addrs=127.6.235.126:40053
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.235.65
--log_dir=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790561432251112-7085-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 00:51:59 UTC on ccycloud.quasar-axgkwn.root.comops.site
build id 4676464
ASAN enabled
I20260928 02:14:14.007786 21566 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:14:14.009343 21566 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:14:14.026011 21591 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:14:14.026016 21592 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:14:14.029798 21594 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:14:14.030699 21566 server_base.cc:1034] running on GCE node
I20260928 02:14:14.031509 21566 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:14:14.032768 21566 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:14:14.034075 21566 hybrid_clock.cc:648] HybridClock initialized: now 1790561654034003 us; error 84 us; skew 500 ppm
I20260928 02:14:14.034473 21566 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 02:14:14 dist-test-slave-86gv krb5kdc[20943](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561654, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.235.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:14:14.038734 21566 init.cc:377] Logged in from keytab as kudu/127.6.235.65@KRBTEST.COM (short username kudu)
I20260928 02:14:14.040660 21566 webserver.cc:460] Webserver started at http://127.6.235.65:41415/ using document root <none> and password file <none>
I20260928 02:14:14.041625 21566 fs_manager.cc:359] Metadata directory not provided
I20260928 02:14:14.041779 21566 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:14:14.049006 21566 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.004s sys 0.001s
I20260928 02:14:14.052565 21601 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:14:14.054157 21566 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260928 02:14:14.054332 21566 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790561432251112-7085-0/minicluster-data/ts-0/data,/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790561432251112-7085-0/minicluster-data/ts-0/wal
uuid: "37070660923a4173b9bfe2c5b3abc0c4"
format_stamp: "Formatted at 2026-09-28 02:14:10 on dist-test-slave-86gv"
server_key: "1b06c11186b1b991199074c31887352b"
server_key_iv: "1f600e3577ba6b34682b7fb5036eb7df"
server_key_version: "encryptionkey@0"
I20260928 02:14:14.055429 21566 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790561432251112-7085-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790561432251112-7085-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790561432251112-7085-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:14:14.075923 21566 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:14:14.080154 21566 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:14:14.080375 21566 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:14:14.081382 21566 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:14:14.082644 21566 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:14:14.082729 21566 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:14:14.082831 21566 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:14:14.082888 21566 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:14:14.117633 21566 rpc_server.cc:307] RPC server started. Bound to: 127.6.235.65:37261
I20260928 02:14:14.117727 21713 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.235.65:37261 every 8 connection(s)
I20260928 02:14:14.119211 21566 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790561432251112-7085-0/minicluster-data/ts-0/data/info.pb
I20260928 02:14:14.125946 7085 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTlaUuc/build/asan/bin/kudu as pid 21566
I20260928 02:14:14.126796 7085 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTlaUuc/build/asan/bin/kudu
/tmp/dist-test-taskTlaUuc/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790561432251112-7085-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790561432251112-7085-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790561432251112-7085-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790561432251112-7085-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.235.66:36697
--local_ip_for_outbound_sockets=127.6.235.66
--tserver_master_addrs=127.6.235.126:40053
--webserver_port=45689
--webserver_interface=127.6.235.66
--keytab_file=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790561432251112-7085-0/krb5kdc/kudu_127.6.235.66.keytab
--principal=kudu/127.6.235.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.235.84:34573
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790561432251112-7085-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790561432251112-7085-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790561432251112-7085-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 28 02:14:14 dist-test-slave-86gv krb5kdc[20943](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561654, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.235.65@KRBTEST.COM for kudu/127.6.235.126@KRBTEST.COM
I20260928 02:14:14.147488 21717 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:14:14.122632 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.235.126:40053 (local address 127.6.235.65:58183)
0928 02:14:14.123605 (+ 973us) negotiation.cc:107] Waiting for socket to connect
0928 02:14:14.123651 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0928 02:14:14.125491 (+ 1840us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:14:14.127286 (+ 1795us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:14:14.127294 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:14:14.127823 (+ 529us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:14:14.128381 (+ 558us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:14:14.128397 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:14:14.130969 (+ 2572us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:14:14.130973 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:14:14.132283 (+ 1310us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:14:14.132296 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:14:14.133864 (+ 1568us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:14:14.133872 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:14:14.134206 (+ 334us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:14:14.135040 (+ 834us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:14:14.135067 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:14:14.137862 (+ 2795us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:14:14.142207 (+ 4345us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:14:14.142220 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:14:14.142238 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:14:14.142920 (+ 682us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:14:14.143561 (+ 641us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:14:14.143566 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:14:14.143570 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:14:14.143816 (+ 246us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:14:14.144790 (+ 974us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:14:14.144798 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:14:14.145203 (+ 405us) client_negotiation.cc:769] Sending connection context
0928 02:14:14.145502 (+ 299us) client_negotiation.cc:240] Negotiation successful
0928 02:14:14.146107 (+ 605us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":651,"thread_start_us":425,"threads_started":1}
I20260928 02:14:14.147495 21716 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:14:14.122372 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.235.65:58183 (local address 127.6.235.126:40053)
0928 02:14:14.123008 (+ 636us) server_negotiation.cc:206] Beginning negotiation
0928 02:14:14.123021 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0928 02:14:14.123784 (+ 763us) server_negotiation.cc:407] Connection header received
0928 02:14:14.126167 (+ 2383us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:14:14.126187 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:14:14.126719 (+ 532us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:14:14.127153 (+ 434us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:14:14.128653 (+ 1500us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:14:14.130783 (+ 2130us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:14:14.132448 (+ 1665us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:14:14.133608 (+ 1160us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:14:14.133808 (+ 200us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:14:14.138178 (+ 4370us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:14:14.138186 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:14:14.138202 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:14:14.138246 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:14:14.142041 (+ 3795us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:14:14.143092 (+ 1051us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:14:14.143102 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:14:14.143111 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:14:14.143378 (+ 267us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:14:14.143994 (+ 616us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:14:14.143999 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:14:14.144003 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:14:14.144636 (+ 633us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:14:14.144854 (+ 218us) server_negotiation.cc:1035] Waiting for connection context
0928 02:14:14.145721 (+ 867us) server_negotiation.cc:299] Negotiation successful
0928 02:14:14.146123 (+ 402us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":432,"thread_start_us":272,"threads_started":1}
I20260928 02:14:14.151438 21714 heartbeater.cc:347] Connected to a master server at 127.6.235.126:40053
I20260928 02:14:14.151788 21714 heartbeater.cc:464] Registering TS with master...
I20260928 02:14:14.152575 21714 heartbeater.cc:511] Master 127.6.235.126:40053 requested a full tablet report, sending...
I20260928 02:14:14.155076 21529 ts_manager.cc:194] Registered new tserver with Master: 37070660923a4173b9bfe2c5b3abc0c4 (127.6.235.65:37261)
I20260928 02:14:14.157128 21529 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.235.65@KRBTEST.COM'} at 127.6.235.65:58183
W20260928 02:14:14.467296 21718 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:14:14.467598 21718 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:14:14.467692 21718 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:14:14.475324 21718 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:14:14.475451 21718 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:14:14.475565 21718 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.235.66
I20260928 02:14:14.485389 21718 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.235.84:34573
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790561432251112-7085-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790561432251112-7085-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790561432251112-7085-0/krb5kdc/kudu_127.6.235.66.keytab
--principal=kudu/127.6.235.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.235.66:36697
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790561432251112-7085-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.6.235.66
--webserver_port=45689
--tserver_master_addrs=127.6.235.126:40053
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.235.66
--log_dir=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790561432251112-7085-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 00:51:59 UTC on ccycloud.quasar-axgkwn.root.comops.site
build id 4676464
ASAN enabled
I20260928 02:14:14.486709 21718 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:14:14.488034 21718 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:14:14.505378 21726 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:14:14.505445 21724 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:14:14.505424 21723 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:14:14.506464 21718 server_base.cc:1034] running on GCE node
I20260928 02:14:14.507179 21718 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:14:14.508244 21718 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:14:14.509491 21718 hybrid_clock.cc:648] HybridClock initialized: now 1790561654509441 us; error 54 us; skew 500 ppm
I20260928 02:14:14.509860 21718 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 02:14:14 dist-test-slave-86gv krb5kdc[20943](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561654, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.235.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:14:14.513295 21718 init.cc:377] Logged in from keytab as kudu/127.6.235.66@KRBTEST.COM (short username kudu)
I20260928 02:14:14.514905 21718 webserver.cc:460] Webserver started at http://127.6.235.66:45689/ using document root <none> and password file <none>
I20260928 02:14:14.515659 21718 fs_manager.cc:359] Metadata directory not provided
I20260928 02:14:14.515774 21718 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:14:14.521765 21718 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20260928 02:14:14.524767 21733 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:14:14.526350 21718 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260928 02:14:14.526503 21718 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790561432251112-7085-0/minicluster-data/ts-1/data,/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790561432251112-7085-0/minicluster-data/ts-1/wal
uuid: "f820529e7c8e4f2f8341f373b1def2d0"
format_stamp: "Formatted at 2026-09-28 02:14:11 on dist-test-slave-86gv"
server_key: "2121432c3e1f641baf637d3823de8d3d"
server_key_iv: "659dd1f1aa6458b3c06d8512e968d2c9"
server_key_version: "encryptionkey@0"
I20260928 02:14:14.527511 21718 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790561432251112-7085-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790561432251112-7085-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790561432251112-7085-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:14:14.552163 21718 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:14:14.556985 21718 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:14:14.557224 21718 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:14:14.558220 21718 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:14:14.559734 21718 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:14:14.559860 21718 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:14:14.560039 21718 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:14:14.560140 21718 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:14:14.594607 21718 rpc_server.cc:307] RPC server started. Bound to: 127.6.235.66:36697
I20260928 02:14:14.594641 21845 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.235.66:36697 every 8 connection(s)
I20260928 02:14:14.596558 21718 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790561432251112-7085-0/minicluster-data/ts-1/data/info.pb
I20260928 02:14:14.604780 7085 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTlaUuc/build/asan/bin/kudu as pid 21718
I20260928 02:14:14.606233 7085 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTlaUuc/build/asan/bin/kudu
/tmp/dist-test-taskTlaUuc/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790561432251112-7085-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790561432251112-7085-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790561432251112-7085-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790561432251112-7085-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.235.67:41399
--local_ip_for_outbound_sockets=127.6.235.67
--tserver_master_addrs=127.6.235.126:40053
--webserver_port=40207
--webserver_interface=127.6.235.67
--keytab_file=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790561432251112-7085-0/krb5kdc/kudu_127.6.235.67.keytab
--principal=kudu/127.6.235.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.235.84:34573
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790561432251112-7085-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790561432251112-7085-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790561432251112-7085-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 28 02:14:14 dist-test-slave-86gv krb5kdc[20943](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561654, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.235.66@KRBTEST.COM for kudu/127.6.235.126@KRBTEST.COM
I20260928 02:14:14.621198 21716 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:14:14.599396 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.235.66:40481 (local address 127.6.235.126:40053)
0928 02:14:14.599585 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0928 02:14:14.599589 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:14:14.601092 (+ 1503us) server_negotiation.cc:407] Connection header received
0928 02:14:14.603035 (+ 1943us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:14:14.603041 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:14:14.603155 (+ 114us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:14:14.603271 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:14:14.605248 (+ 1977us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:14:14.606064 (+ 816us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:14:14.609048 (+ 2984us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:14:14.609519 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:14:14.609632 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:14:14.614585 (+ 4953us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:14:14.614594 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:14:14.614601 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:14:14.614669 (+ 68us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:14:14.617498 (+ 2829us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:14:14.618364 (+ 866us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:14:14.618369 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:14:14.618371 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:14:14.618558 (+ 187us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:14:14.619150 (+ 592us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:14:14.619153 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:14:14.619156 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:14:14.619440 (+ 284us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:14:14.619554 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0928 02:14:14.620595 (+ 1041us) server_negotiation.cc:299] Negotiation successful
0928 02:14:14.620871 (+ 276us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260928 02:14:14.622745 21848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:14:14.600071 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.235.126:40053 (local address 127.6.235.66:40481)
0928 02:14:14.600851 (+ 780us) negotiation.cc:107] Waiting for socket to connect
0928 02:14:14.600969 (+ 118us) client_negotiation.cc:174] Beginning negotiation
0928 02:14:14.602724 (+ 1755us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:14:14.603545 (+ 821us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:14:14.603554 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:14:14.604107 (+ 553us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:14:14.605020 (+ 913us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:14:14.605053 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:14:14.606305 (+ 1252us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:14:14.606311 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:14:14.608650 (+ 2339us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:14:14.608672 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:14:14.609803 (+ 1131us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:14:14.609813 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:14:14.610178 (+ 365us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:14:14.611108 (+ 930us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:14:14.611150 (+ 42us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:14:14.614312 (+ 3162us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:14:14.617675 (+ 3363us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:14:14.617685 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:14:14.617706 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:14:14.618213 (+ 507us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:14:14.618715 (+ 502us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:14:14.618724 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:14:14.618730 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:14:14.619016 (+ 286us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:14:14.619574 (+ 558us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:14:14.619585 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:14:14.620102 (+ 517us) client_negotiation.cc:769] Sending connection context
0928 02:14:14.620492 (+ 390us) client_negotiation.cc:240] Negotiation successful
0928 02:14:14.621162 (+ 670us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":521,"thread_start_us":320,"threads_started":1}
I20260928 02:14:14.624475 21846 heartbeater.cc:347] Connected to a master server at 127.6.235.126:40053
I20260928 02:14:14.624825 21846 heartbeater.cc:464] Registering TS with master...
I20260928 02:14:14.625808 21846 heartbeater.cc:511] Master 127.6.235.126:40053 requested a full tablet report, sending...
I20260928 02:14:14.627655 21529 ts_manager.cc:194] Registered new tserver with Master: f820529e7c8e4f2f8341f373b1def2d0 (127.6.235.66:36697)
I20260928 02:14:14.628532 21529 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.235.66@KRBTEST.COM'} at 127.6.235.66:40481
W20260928 02:14:14.959913 21849 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:14:14.960265 21849 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:14:14.960464 21849 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:14:14.968111 21849 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:14:14.968235 21849 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:14:14.968418 21849 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.235.67
I20260928 02:14:14.978721 21849 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.235.84:34573
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790561432251112-7085-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790561432251112-7085-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790561432251112-7085-0/krb5kdc/kudu_127.6.235.67.keytab
--principal=kudu/127.6.235.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.235.67:41399
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790561432251112-7085-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.6.235.67
--webserver_port=40207
--tserver_master_addrs=127.6.235.126:40053
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.235.67
--log_dir=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790561432251112-7085-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 00:51:59 UTC on ccycloud.quasar-axgkwn.root.comops.site
build id 4676464
ASAN enabled
I20260928 02:14:14.980144 21849 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:14:14.981702 21849 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:14:14.996714 21857 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:14:14.996626 21855 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:14:14.996755 21854 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:14:14.997486 21849 server_base.cc:1034] running on GCE node
I20260928 02:14:14.998332 21849 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:14:14.999440 21849 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:14:15.000727 21849 hybrid_clock.cc:648] HybridClock initialized: now 1790561655000676 us; error 59 us; skew 500 ppm
I20260928 02:14:15.001155 21849 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 02:14:14 dist-test-slave-86gv krb5kdc[20943](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561654, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.235.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:14:15.004496 21849 init.cc:377] Logged in from keytab as kudu/127.6.235.67@KRBTEST.COM (short username kudu)
I20260928 02:14:15.006402 21849 webserver.cc:460] Webserver started at http://127.6.235.67:40207/ using document root <none> and password file <none>
I20260928 02:14:15.007180 21849 fs_manager.cc:359] Metadata directory not provided
I20260928 02:14:15.007345 21849 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:14:15.013293 21849 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.001s
I20260928 02:14:15.016462 21864 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:14:15.018131 21849 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 02:14:15.018319 21849 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790561432251112-7085-0/minicluster-data/ts-2/data,/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790561432251112-7085-0/minicluster-data/ts-2/wal
uuid: "f11b50d0630c4d91be9b17973fcb0230"
format_stamp: "Formatted at 2026-09-28 02:14:11 on dist-test-slave-86gv"
server_key: "51fe6d056371e4b2efa727b487ae6e62"
server_key_iv: "32e2a67cf03cccfcde49d183927eee55"
server_key_version: "encryptionkey@0"
I20260928 02:14:15.019336 21849 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790561432251112-7085-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790561432251112-7085-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790561432251112-7085-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:14:15.035643 21849 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:14:15.040210 21849 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:14:15.040511 21849 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:14:15.041594 21849 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:14:15.043076 21849 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:14:15.043198 21849 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:14:15.043336 21849 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:14:15.043427 21849 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:14:15.078097 21849 rpc_server.cc:307] RPC server started. Bound to: 127.6.235.67:41399
I20260928 02:14:15.078186 21976 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.235.67:41399 every 8 connection(s)
I20260928 02:14:15.079602 21849 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790561432251112-7085-0/minicluster-data/ts-2/data/info.pb
I20260928 02:14:15.085808 7085 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTlaUuc/build/asan/bin/kudu as pid 21849
Sep 28 02:14:15 dist-test-slave-86gv krb5kdc[20943](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561654, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.235.67@KRBTEST.COM for kudu/127.6.235.126@KRBTEST.COM
I20260928 02:14:15.103016 21980 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:14:15.086848 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35022 (local address 127.6.235.126:40053)
0928 02:14:15.087320 (+ 472us) server_negotiation.cc:206] Beginning negotiation
0928 02:14:15.087325 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 02:14:15.087699 (+ 374us) server_negotiation.cc:407] Connection header received
0928 02:14:15.087831 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:14:15.087838 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:14:15.087924 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:14:15.088020 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:14:15.089900 (+ 1880us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:14:15.091009 (+ 1109us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:14:15.092976 (+ 1967us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:14:15.093665 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:14:15.093791 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:14:15.096594 (+ 2803us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:14:15.096599 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:14:15.096602 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:14:15.096675 (+ 73us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:14:15.099957 (+ 3282us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:14:15.100787 (+ 830us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:14:15.100792 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:14:15.100795 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:14:15.100970 (+ 175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:14:15.101553 (+ 583us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:14:15.101556 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:14:15.101558 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:14:15.101862 (+ 304us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:14:15.102096 (+ 234us) server_negotiation.cc:1035] Waiting for connection context
0928 02:14:15.102360 (+ 264us) server_negotiation.cc:299] Negotiation successful
0928 02:14:15.102604 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":365,"thread_start_us":251,"threads_started":1}
I20260928 02:14:15.103441 21716 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:14:15.082337 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.235.67:46461 (local address 127.6.235.126:40053)
0928 02:14:15.082546 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0928 02:14:15.082550 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:14:15.083736 (+ 1186us) server_negotiation.cc:407] Connection header received
0928 02:14:15.085766 (+ 2030us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:14:15.085770 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:14:15.085840 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:14:15.085984 (+ 144us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:14:15.087929 (+ 1945us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:14:15.089155 (+ 1226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:14:15.091318 (+ 2163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:14:15.091968 (+ 650us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:14:15.092095 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:14:15.095859 (+ 3764us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:14:15.095867 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:14:15.095873 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:14:15.095928 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:14:15.099347 (+ 3419us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:14:15.100282 (+ 935us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:14:15.100287 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:14:15.100291 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:14:15.100449 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:14:15.101066 (+ 617us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:14:15.101074 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:14:15.101078 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:14:15.101694 (+ 616us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:14:15.101835 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0928 02:14:15.102808 (+ 973us) server_negotiation.cc:299] Negotiation successful
0928 02:14:15.103009 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
I20260928 02:14:15.104504 21979 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:14:15.082866 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.235.126:40053 (local address 127.6.235.67:46461)
0928 02:14:15.083567 (+ 701us) negotiation.cc:107] Waiting for socket to connect
0928 02:14:15.083611 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0928 02:14:15.085430 (+ 1819us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:14:15.086242 (+ 812us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:14:15.086255 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:14:15.086933 (+ 678us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:14:15.087709 (+ 776us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:14:15.087738 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:14:15.089330 (+ 1592us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:14:15.089334 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:14:15.091188 (+ 1854us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:14:15.091199 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:14:15.092173 (+ 974us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:14:15.092177 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:14:15.092376 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:14:15.093160 (+ 784us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:14:15.093189 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:14:15.095624 (+ 2435us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:14:15.099499 (+ 3875us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:14:15.099511 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:14:15.099531 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:14:15.100167 (+ 636us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:14:15.100562 (+ 395us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:14:15.100565 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:14:15.100567 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:14:15.100783 (+ 216us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:14:15.101839 (+ 1056us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:14:15.101847 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:14:15.102329 (+ 482us) client_negotiation.cc:769] Sending connection context
0928 02:14:15.102658 (+ 329us) client_negotiation.cc:240] Negotiation successful
0928 02:14:15.103223 (+ 565us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":452,"thread_start_us":265,"threads_started":1}
I20260928 02:14:15.106086 21977 heartbeater.cc:347] Connected to a master server at 127.6.235.126:40053
I20260928 02:14:15.106456 21977 heartbeater.cc:464] Registering TS with master...
I20260928 02:14:15.107242 21977 heartbeater.cc:511] Master 127.6.235.126:40053 requested a full tablet report, sending...
I20260928 02:14:15.108822 21528 ts_manager.cc:194] Registered new tserver with Master: f11b50d0630c4d91be9b17973fcb0230 (127.6.235.67:41399)
I20260928 02:14:15.109773 21528 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.235.67@KRBTEST.COM'} at 127.6.235.67:46461
I20260928 02:14:15.118361 7085 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 02:14:15.125871 21716 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:14:15.119351 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35034 (local address 127.6.235.126:40053)
0928 02:14:15.119568 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0928 02:14:15.119576 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 02:14:15.120039 (+ 463us) server_negotiation.cc:407] Connection header received
0928 02:14:15.120242 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:14:15.120245 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:14:15.120372 (+ 127us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:14:15.120494 (+ 122us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 02:14:15.121036 (+ 542us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:14:15.121768 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:14:15.123738 (+ 1970us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:14:15.124256 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:14:15.124348 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:14:15.124779 (+ 431us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 02:14:15.125309 (+ 530us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 02:14:15.125415 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0928 02:14:15.125508 (+ 93us) server_negotiation.cc:299] Negotiation successful
0928 02:14:15.125572 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260928 02:14:15.126811 21528 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin'} at 127.0.0.1:35034:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 02:14:15.134157 21528 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 02:14:15.160602 21714 heartbeater.cc:503] Master 127.6.235.126:40053 was elected leader, sending a full tablet report...
I20260928 02:14:15.172011 21990 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:14:15.148206 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48334 (local address 127.6.235.66:36697)
0928 02:14:15.163989 (+ 15783us) server_negotiation.cc:206] Beginning negotiation
0928 02:14:15.163998 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 02:14:15.164030 (+ 32us) server_negotiation.cc:407] Connection header received
0928 02:14:15.164129 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:14:15.164138 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:14:15.164453 (+ 315us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:14:15.164699 (+ 246us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:14:15.165338 (+ 639us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:14:15.166757 (+ 1419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:14:15.168834 (+ 2077us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:14:15.170447 (+ 1613us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:14:15.170578 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:14:15.171292 (+ 714us) server_negotiation.cc:1035] Waiting for connection context
0928 02:14:15.171438 (+ 146us) server_negotiation.cc:299] Negotiation successful
0928 02:14:15.171637 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15595,"thread_start_us":15484,"threads_started":1}
I20260928 02:14:15.172170 21989 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:14:15.147660 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.235.66:36697 (local address 127.0.0.1:48334)
0928 02:14:15.158058 (+ 10398us) negotiation.cc:107] Waiting for socket to connect
0928 02:14:15.158096 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0928 02:14:15.158304 (+ 208us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:14:15.164684 (+ 6380us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:14:15.164689 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:14:15.164717 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:14:15.165154 (+ 437us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:14:15.165163 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:14:15.166903 (+ 1740us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:14:15.166909 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:14:15.168666 (+ 1757us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:14:15.168678 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:14:15.170909 (+ 2231us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:14:15.170916 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:14:15.171098 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:14:15.171235 (+ 137us) client_negotiation.cc:769] Sending connection context
0928 02:14:15.171392 (+ 157us) client_negotiation.cc:240] Negotiation successful
0928 02:14:15.171682 (+ 290us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10176,"mutex_wait_us":134,"thread_start_us":10784,"threads_started":1}
I20260928 02:14:15.172297 21987 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:14:15.147012 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.235.67:41399 (local address 127.0.0.1:58866)
0928 02:14:15.148194 (+ 1182us) negotiation.cc:107] Waiting for socket to connect
0928 02:14:15.148239 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0928 02:14:15.148440 (+ 201us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:14:15.160799 (+ 12359us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:14:15.160812 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:14:15.160863 (+ 51us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:14:15.161481 (+ 618us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:14:15.161501 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:14:15.164574 (+ 3073us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:14:15.164584 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:14:15.168218 (+ 3634us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:14:15.168233 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:14:15.171072 (+ 2839us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:14:15.171080 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:14:15.171267 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:14:15.171313 (+ 46us) client_negotiation.cc:769] Sending connection context
0928 02:14:15.171425 (+ 112us) client_negotiation.cc:240] Negotiation successful
0928 02:14:15.171682 (+ 257us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":906,"mutex_wait_us":37,"thread_start_us":836,"threads_started":1}
I20260928 02:14:15.173167 21988 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:14:15.147670 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58866 (local address 127.6.235.67:41399)
0928 02:14:15.159374 (+ 11704us) server_negotiation.cc:206] Beginning negotiation
0928 02:14:15.159388 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0928 02:14:15.159532 (+ 144us) server_negotiation.cc:407] Connection header received
0928 02:14:15.159674 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:14:15.159686 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:14:15.160058 (+ 372us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:14:15.160337 (+ 279us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:14:15.162644 (+ 2307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:14:15.164354 (+ 1710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:14:15.168469 (+ 4115us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:14:15.170354 (+ 1885us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:14:15.170528 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:14:15.170808 (+ 280us) server_negotiation.cc:1035] Waiting for connection context
0928 02:14:15.172622 (+ 1814us) server_negotiation.cc:299] Negotiation successful
0928 02:14:15.172820 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11492,"thread_start_us":11335,"threads_started":1}
I20260928 02:14:15.176244 21991 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:14:15.148870 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.235.65:37261 (local address 127.0.0.1:47550)
0928 02:14:15.163190 (+ 14320us) negotiation.cc:107] Waiting for socket to connect
0928 02:14:15.163210 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0928 02:14:15.163323 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:14:15.168078 (+ 4755us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:14:15.168084 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:14:15.168117 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:14:15.168588 (+ 471us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:14:15.168600 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:14:15.170581 (+ 1981us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:14:15.170586 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:14:15.173351 (+ 2765us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:14:15.173368 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:14:15.175263 (+ 1895us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:14:15.175270 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:14:15.175631 (+ 361us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:14:15.175667 (+ 36us) client_negotiation.cc:769] Sending connection context
0928 02:14:15.175735 (+ 68us) client_negotiation.cc:240] Negotiation successful
0928 02:14:15.175867 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14187,"mutex_wait_us":28,"thread_start_us":14489,"threads_started":1}
I20260928 02:14:15.176613 21912 tablet_service.cc:1467] Processing CreateTablet for tablet b6e74e850a0e463f8677dd6a1e1982c4 (DEFAULT_TABLE table=test-table [id=7e8ba4d0c122435098a76188625b2db9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:14:15.176630 21781 tablet_service.cc:1467] Processing CreateTablet for tablet b6e74e850a0e463f8677dd6a1e1982c4 (DEFAULT_TABLE table=test-table [id=7e8ba4d0c122435098a76188625b2db9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:14:15.177438 21992 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:14:15.149332 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47550 (local address 127.6.235.65:37261)
0928 02:14:15.167387 (+ 18055us) server_negotiation.cc:206] Beginning negotiation
0928 02:14:15.167396 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 02:14:15.167426 (+ 30us) server_negotiation.cc:407] Connection header received
0928 02:14:15.167527 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:14:15.167536 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:14:15.167828 (+ 292us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:14:15.168061 (+ 233us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:14:15.168770 (+ 709us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:14:15.170434 (+ 1664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:14:15.173573 (+ 3139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:14:15.175009 (+ 1436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:14:15.176610 (+ 1601us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:14:15.176849 (+ 239us) server_negotiation.cc:1035] Waiting for connection context
0928 02:14:15.176953 (+ 104us) server_negotiation.cc:299] Negotiation successful
0928 02:14:15.177122 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17871,"thread_start_us":18919,"threads_started":1}
I20260928 02:14:15.178048 21912 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b6e74e850a0e463f8677dd6a1e1982c4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:14:15.178097 21781 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b6e74e850a0e463f8677dd6a1e1982c4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:14:15.181313 21649 tablet_service.cc:1467] Processing CreateTablet for tablet b6e74e850a0e463f8677dd6a1e1982c4 (DEFAULT_TABLE table=test-table [id=7e8ba4d0c122435098a76188625b2db9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:14:15.182847 21649 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b6e74e850a0e463f8677dd6a1e1982c4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:14:15.190995 21993 tablet_bootstrap.cc:492] T b6e74e850a0e463f8677dd6a1e1982c4 P f11b50d0630c4d91be9b17973fcb0230: Bootstrap starting.
I20260928 02:14:15.194453 21993 tablet_bootstrap.cc:654] T b6e74e850a0e463f8677dd6a1e1982c4 P f11b50d0630c4d91be9b17973fcb0230: Neither blocks nor log segments found. Creating new log.
I20260928 02:14:15.194610 21994 tablet_bootstrap.cc:492] T b6e74e850a0e463f8677dd6a1e1982c4 P f820529e7c8e4f2f8341f373b1def2d0: Bootstrap starting.
I20260928 02:14:15.195986 21993 log.cc:824] T b6e74e850a0e463f8677dd6a1e1982c4 P f11b50d0630c4d91be9b17973fcb0230: Log is configured to *not* fsync() on all Append() calls
I20260928 02:14:15.198290 21994 tablet_bootstrap.cc:654] T b6e74e850a0e463f8677dd6a1e1982c4 P f820529e7c8e4f2f8341f373b1def2d0: Neither blocks nor log segments found. Creating new log.
I20260928 02:14:15.198355 21996 tablet_bootstrap.cc:492] T b6e74e850a0e463f8677dd6a1e1982c4 P 37070660923a4173b9bfe2c5b3abc0c4: Bootstrap starting.
I20260928 02:14:15.199837 21994 log.cc:824] T b6e74e850a0e463f8677dd6a1e1982c4 P f820529e7c8e4f2f8341f373b1def2d0: Log is configured to *not* fsync() on all Append() calls
I20260928 02:14:15.201050 21993 tablet_bootstrap.cc:492] T b6e74e850a0e463f8677dd6a1e1982c4 P f11b50d0630c4d91be9b17973fcb0230: No bootstrap required, opened a new log
I20260928 02:14:15.201368 21993 ts_tablet_manager.cc:1397] T b6e74e850a0e463f8677dd6a1e1982c4 P f11b50d0630c4d91be9b17973fcb0230: Time spent bootstrapping tablet: real 0.011s user 0.008s sys 0.000s
I20260928 02:14:15.201851 21996 tablet_bootstrap.cc:654] T b6e74e850a0e463f8677dd6a1e1982c4 P 37070660923a4173b9bfe2c5b3abc0c4: Neither blocks nor log segments found. Creating new log.
I20260928 02:14:15.202843 21996 log.cc:824] T b6e74e850a0e463f8677dd6a1e1982c4 P 37070660923a4173b9bfe2c5b3abc0c4: Log is configured to *not* fsync() on all Append() calls
I20260928 02:14:15.203208 21994 tablet_bootstrap.cc:492] T b6e74e850a0e463f8677dd6a1e1982c4 P f820529e7c8e4f2f8341f373b1def2d0: No bootstrap required, opened a new log
I20260928 02:14:15.203487 21994 ts_tablet_manager.cc:1397] T b6e74e850a0e463f8677dd6a1e1982c4 P f820529e7c8e4f2f8341f373b1def2d0: Time spent bootstrapping tablet: real 0.009s user 0.000s sys 0.006s
I20260928 02:14:15.205694 21996 tablet_bootstrap.cc:492] T b6e74e850a0e463f8677dd6a1e1982c4 P 37070660923a4173b9bfe2c5b3abc0c4: No bootstrap required, opened a new log
I20260928 02:14:15.205971 21996 ts_tablet_manager.cc:1397] T b6e74e850a0e463f8677dd6a1e1982c4 P 37070660923a4173b9bfe2c5b3abc0c4: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260928 02:14:15.212604 21993 raft_consensus.cc:348] T b6e74e850a0e463f8677dd6a1e1982c4 P f11b50d0630c4d91be9b17973fcb0230 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f11b50d0630c4d91be9b17973fcb0230" member_type: VOTER last_known_addr { host: "127.6.235.67" port: 41399 } } peers { permanent_uuid: "37070660923a4173b9bfe2c5b3abc0c4" member_type: VOTER last_known_addr { host: "127.6.235.65" port: 37261 } } peers { permanent_uuid: "f820529e7c8e4f2f8341f373b1def2d0" member_type: VOTER last_known_addr { host: "127.6.235.66" port: 36697 } }
I20260928 02:14:15.213191 21993 raft_consensus.cc:374] T b6e74e850a0e463f8677dd6a1e1982c4 P f11b50d0630c4d91be9b17973fcb0230 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:14:15.213302 21993 raft_consensus.cc:729] T b6e74e850a0e463f8677dd6a1e1982c4 P f11b50d0630c4d91be9b17973fcb0230 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f11b50d0630c4d91be9b17973fcb0230, State: Initialized, Role: FOLLOWER
I20260928 02:14:15.214004 21993 consensus_queue.cc:260] T b6e74e850a0e463f8677dd6a1e1982c4 P f11b50d0630c4d91be9b17973fcb0230 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f11b50d0630c4d91be9b17973fcb0230" member_type: VOTER last_known_addr { host: "127.6.235.67" port: 41399 } } peers { permanent_uuid: "37070660923a4173b9bfe2c5b3abc0c4" member_type: VOTER last_known_addr { host: "127.6.235.65" port: 37261 } } peers { permanent_uuid: "f820529e7c8e4f2f8341f373b1def2d0" member_type: VOTER last_known_addr { host: "127.6.235.66" port: 36697 } }
I20260928 02:14:15.213994 21996 raft_consensus.cc:348] T b6e74e850a0e463f8677dd6a1e1982c4 P 37070660923a4173b9bfe2c5b3abc0c4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f11b50d0630c4d91be9b17973fcb0230" member_type: VOTER last_known_addr { host: "127.6.235.67" port: 41399 } } peers { permanent_uuid: "37070660923a4173b9bfe2c5b3abc0c4" member_type: VOTER last_known_addr { host: "127.6.235.65" port: 37261 } } peers { permanent_uuid: "f820529e7c8e4f2f8341f373b1def2d0" member_type: VOTER last_known_addr { host: "127.6.235.66" port: 36697 } }
I20260928 02:14:15.214454 21996 raft_consensus.cc:374] T b6e74e850a0e463f8677dd6a1e1982c4 P 37070660923a4173b9bfe2c5b3abc0c4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:14:15.214573 21996 raft_consensus.cc:729] T b6e74e850a0e463f8677dd6a1e1982c4 P 37070660923a4173b9bfe2c5b3abc0c4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 37070660923a4173b9bfe2c5b3abc0c4, State: Initialized, Role: FOLLOWER
I20260928 02:14:15.214289 21994 raft_consensus.cc:348] T b6e74e850a0e463f8677dd6a1e1982c4 P f820529e7c8e4f2f8341f373b1def2d0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f11b50d0630c4d91be9b17973fcb0230" member_type: VOTER last_known_addr { host: "127.6.235.67" port: 41399 } } peers { permanent_uuid: "37070660923a4173b9bfe2c5b3abc0c4" member_type: VOTER last_known_addr { host: "127.6.235.65" port: 37261 } } peers { permanent_uuid: "f820529e7c8e4f2f8341f373b1def2d0" member_type: VOTER last_known_addr { host: "127.6.235.66" port: 36697 } }
I20260928 02:14:15.214767 21994 raft_consensus.cc:374] T b6e74e850a0e463f8677dd6a1e1982c4 P f820529e7c8e4f2f8341f373b1def2d0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:14:15.214890 21994 raft_consensus.cc:729] T b6e74e850a0e463f8677dd6a1e1982c4 P f820529e7c8e4f2f8341f373b1def2d0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f820529e7c8e4f2f8341f373b1def2d0, State: Initialized, Role: FOLLOWER
I20260928 02:14:15.215677 21994 consensus_queue.cc:260] T b6e74e850a0e463f8677dd6a1e1982c4 P f820529e7c8e4f2f8341f373b1def2d0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f11b50d0630c4d91be9b17973fcb0230" member_type: VOTER last_known_addr { host: "127.6.235.67" port: 41399 } } peers { permanent_uuid: "37070660923a4173b9bfe2c5b3abc0c4" member_type: VOTER last_known_addr { host: "127.6.235.65" port: 37261 } } peers { permanent_uuid: "f820529e7c8e4f2f8341f373b1def2d0" member_type: VOTER last_known_addr { host: "127.6.235.66" port: 36697 } }
I20260928 02:14:15.217231 21846 heartbeater.cc:503] Master 127.6.235.126:40053 was elected leader, sending a full tablet report...
I20260928 02:14:15.217950 21994 ts_tablet_manager.cc:1428] T b6e74e850a0e463f8677dd6a1e1982c4 P f820529e7c8e4f2f8341f373b1def2d0: Time spent starting tablet: real 0.014s user 0.011s sys 0.005s
I20260928 02:14:15.218926 21977 heartbeater.cc:503] Master 127.6.235.126:40053 was elected leader, sending a full tablet report...
I20260928 02:14:15.219862 21993 ts_tablet_manager.cc:1428] T b6e74e850a0e463f8677dd6a1e1982c4 P f11b50d0630c4d91be9b17973fcb0230: Time spent starting tablet: real 0.018s user 0.013s sys 0.003s
I20260928 02:14:15.219877 21996 consensus_queue.cc:260] T b6e74e850a0e463f8677dd6a1e1982c4 P 37070660923a4173b9bfe2c5b3abc0c4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f11b50d0630c4d91be9b17973fcb0230" member_type: VOTER last_known_addr { host: "127.6.235.67" port: 41399 } } peers { permanent_uuid: "37070660923a4173b9bfe2c5b3abc0c4" member_type: VOTER last_known_addr { host: "127.6.235.65" port: 37261 } } peers { permanent_uuid: "f820529e7c8e4f2f8341f373b1def2d0" member_type: VOTER last_known_addr { host: "127.6.235.66" port: 36697 } }
I20260928 02:14:15.220805 22000 raft_consensus.cc:482] T b6e74e850a0e463f8677dd6a1e1982c4 P 37070660923a4173b9bfe2c5b3abc0c4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 02:14:15.221182 22000 raft_consensus.cc:504] T b6e74e850a0e463f8677dd6a1e1982c4 P 37070660923a4173b9bfe2c5b3abc0c4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f11b50d0630c4d91be9b17973fcb0230" member_type: VOTER last_known_addr { host: "127.6.235.67" port: 41399 } } peers { permanent_uuid: "37070660923a4173b9bfe2c5b3abc0c4" member_type: VOTER last_known_addr { host: "127.6.235.65" port: 37261 } } peers { permanent_uuid: "f820529e7c8e4f2f8341f373b1def2d0" member_type: VOTER last_known_addr { host: "127.6.235.66" port: 36697 } }
I20260928 02:14:15.224087 22000 leader_election.cc:288] T b6e74e850a0e463f8677dd6a1e1982c4 P 37070660923a4173b9bfe2c5b3abc0c4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f11b50d0630c4d91be9b17973fcb0230 (127.6.235.67:41399), f820529e7c8e4f2f8341f373b1def2d0 (127.6.235.66:36697)
I20260928 02:14:15.225492 21996 ts_tablet_manager.cc:1428] T b6e74e850a0e463f8677dd6a1e1982c4 P 37070660923a4173b9bfe2c5b3abc0c4: Time spent starting tablet: real 0.019s user 0.007s sys 0.009s
I20260928 02:14:15.232772 21988 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:14:15.224805 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.235.65:46559 (local address 127.6.235.67:41399)
0928 02:14:15.225013 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0928 02:14:15.225020 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 02:14:15.225180 (+ 160us) server_negotiation.cc:407] Connection header received
0928 02:14:15.225499 (+ 319us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:14:15.225504 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:14:15.225581 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:14:15.225645 (+ 64us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:14:15.226611 (+ 966us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:14:15.227727 (+ 1116us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:14:15.230487 (+ 2760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:14:15.231540 (+ 1053us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:14:15.231673 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:14:15.231730 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0928 02:14:15.232325 (+ 595us) server_negotiation.cc:299] Negotiation successful
0928 02:14:15.232475 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260928 02:14:15.232978 22004 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:14:15.224553 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.235.66:36697 (local address 127.6.235.65:54825)
0928 02:14:15.225412 (+ 859us) negotiation.cc:107] Waiting for socket to connect
0928 02:14:15.225430 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0928 02:14:15.225713 (+ 283us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:14:15.226295 (+ 582us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:14:15.226300 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:14:15.226331 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:14:15.226688 (+ 357us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:14:15.226698 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:14:15.228251 (+ 1553us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:14:15.228257 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:14:15.230375 (+ 2118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:14:15.230389 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:14:15.232030 (+ 1641us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:14:15.232036 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:14:15.232214 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:14:15.232273 (+ 59us) client_negotiation.cc:769] Sending connection context
0928 02:14:15.232363 (+ 90us) client_negotiation.cc:240] Negotiation successful
0928 02:14:15.232501 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":737,"mutex_wait_us":31,"thread_start_us":587,"threads_started":1}
I20260928 02:14:15.233048 22003 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:14:15.224406 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.235.67:41399 (local address 127.6.235.65:46559)
0928 02:14:15.225071 (+ 665us) negotiation.cc:107] Waiting for socket to connect
0928 02:14:15.225095 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0928 02:14:15.225309 (+ 214us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:14:15.225879 (+ 570us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:14:15.225884 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:14:15.225912 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:14:15.226332 (+ 420us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:14:15.226343 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:14:15.227867 (+ 1524us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:14:15.227871 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:14:15.230277 (+ 2406us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:14:15.230295 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:14:15.231761 (+ 1466us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:14:15.231767 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:14:15.232018 (+ 251us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:14:15.232121 (+ 103us) client_negotiation.cc:769] Sending connection context
0928 02:14:15.232254 (+ 133us) client_negotiation.cc:240] Negotiation successful
0928 02:14:15.232484 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":512,"mutex_wait_us":23,"thread_start_us":1760,"threads_started":1}
I20260928 02:14:15.233115 21990 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:14:15.224653 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.235.65:54825 (local address 127.6.235.66:36697)
0928 02:14:15.224933 (+ 280us) server_negotiation.cc:206] Beginning negotiation
0928 02:14:15.224943 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 02:14:15.225519 (+ 576us) server_negotiation.cc:407] Connection header received
0928 02:14:15.225928 (+ 409us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:14:15.225935 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:14:15.226021 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:14:15.226120 (+ 99us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:14:15.227066 (+ 946us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:14:15.228106 (+ 1040us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:14:15.230591 (+ 2485us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:14:15.231788 (+ 1197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:14:15.231954 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:14:15.232028 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0928 02:14:15.232498 (+ 470us) server_negotiation.cc:299] Negotiation successful
0928 02:14:15.232707 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260928 02:14:15.234061 21801 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b6e74e850a0e463f8677dd6a1e1982c4" candidate_uuid: "37070660923a4173b9bfe2c5b3abc0c4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f820529e7c8e4f2f8341f373b1def2d0" is_pre_election: true
I20260928 02:14:15.234377 21932 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b6e74e850a0e463f8677dd6a1e1982c4" candidate_uuid: "37070660923a4173b9bfe2c5b3abc0c4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f11b50d0630c4d91be9b17973fcb0230" is_pre_election: true
I20260928 02:14:15.234597 21801 raft_consensus.cc:2454] T b6e74e850a0e463f8677dd6a1e1982c4 P f820529e7c8e4f2f8341f373b1def2d0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 37070660923a4173b9bfe2c5b3abc0c4 in term 0.
I20260928 02:14:15.234899 21932 raft_consensus.cc:2454] T b6e74e850a0e463f8677dd6a1e1982c4 P f11b50d0630c4d91be9b17973fcb0230 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 37070660923a4173b9bfe2c5b3abc0c4 in term 0.
I20260928 02:14:15.235407 21603 leader_election.cc:302] T b6e74e850a0e463f8677dd6a1e1982c4 P 37070660923a4173b9bfe2c5b3abc0c4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 37070660923a4173b9bfe2c5b3abc0c4, f820529e7c8e4f2f8341f373b1def2d0; no voters:
I20260928 02:14:15.235893 22000 raft_consensus.cc:2781] T b6e74e850a0e463f8677dd6a1e1982c4 P 37070660923a4173b9bfe2c5b3abc0c4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 02:14:15.236071 22000 raft_consensus.cc:482] T b6e74e850a0e463f8677dd6a1e1982c4 P 37070660923a4173b9bfe2c5b3abc0c4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 02:14:15.236191 22000 raft_consensus.cc:3037] T b6e74e850a0e463f8677dd6a1e1982c4 P 37070660923a4173b9bfe2c5b3abc0c4 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:14:15.238538 22000 raft_consensus.cc:504] T b6e74e850a0e463f8677dd6a1e1982c4 P 37070660923a4173b9bfe2c5b3abc0c4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f11b50d0630c4d91be9b17973fcb0230" member_type: VOTER last_known_addr { host: "127.6.235.67" port: 41399 } } peers { permanent_uuid: "37070660923a4173b9bfe2c5b3abc0c4" member_type: VOTER last_known_addr { host: "127.6.235.65" port: 37261 } } peers { permanent_uuid: "f820529e7c8e4f2f8341f373b1def2d0" member_type: VOTER last_known_addr { host: "127.6.235.66" port: 36697 } }
I20260928 02:14:15.239200 22000 leader_election.cc:288] T b6e74e850a0e463f8677dd6a1e1982c4 P 37070660923a4173b9bfe2c5b3abc0c4 [CANDIDATE]: Term 1 election: Requested vote from peers f11b50d0630c4d91be9b17973fcb0230 (127.6.235.67:41399), f820529e7c8e4f2f8341f373b1def2d0 (127.6.235.66:36697)
I20260928 02:14:15.239830 21801 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b6e74e850a0e463f8677dd6a1e1982c4" candidate_uuid: "37070660923a4173b9bfe2c5b3abc0c4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f820529e7c8e4f2f8341f373b1def2d0"
I20260928 02:14:15.239831 21932 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b6e74e850a0e463f8677dd6a1e1982c4" candidate_uuid: "37070660923a4173b9bfe2c5b3abc0c4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f11b50d0630c4d91be9b17973fcb0230"
I20260928 02:14:15.240078 21932 raft_consensus.cc:3037] T b6e74e850a0e463f8677dd6a1e1982c4 P f11b50d0630c4d91be9b17973fcb0230 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:14:15.240073 21801 raft_consensus.cc:3037] T b6e74e850a0e463f8677dd6a1e1982c4 P f820529e7c8e4f2f8341f373b1def2d0 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:14:15.243522 21932 raft_consensus.cc:2454] T b6e74e850a0e463f8677dd6a1e1982c4 P f11b50d0630c4d91be9b17973fcb0230 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 37070660923a4173b9bfe2c5b3abc0c4 in term 1.
I20260928 02:14:15.243522 21801 raft_consensus.cc:2454] T b6e74e850a0e463f8677dd6a1e1982c4 P f820529e7c8e4f2f8341f373b1def2d0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 37070660923a4173b9bfe2c5b3abc0c4 in term 1.
I20260928 02:14:15.244012 21603 leader_election.cc:302] T b6e74e850a0e463f8677dd6a1e1982c4 P 37070660923a4173b9bfe2c5b3abc0c4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 37070660923a4173b9bfe2c5b3abc0c4, f820529e7c8e4f2f8341f373b1def2d0; no voters:
I20260928 02:14:15.244451 22000 raft_consensus.cc:2781] T b6e74e850a0e463f8677dd6a1e1982c4 P 37070660923a4173b9bfe2c5b3abc0c4 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:14:15.244645 22000 raft_consensus.cc:686] T b6e74e850a0e463f8677dd6a1e1982c4 P 37070660923a4173b9bfe2c5b3abc0c4 [term 1 LEADER]: Becoming Leader. State: Replica: 37070660923a4173b9bfe2c5b3abc0c4, State: Running, Role: LEADER
I20260928 02:14:15.245232 22000 consensus_queue.cc:237] T b6e74e850a0e463f8677dd6a1e1982c4 P 37070660923a4173b9bfe2c5b3abc0c4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f11b50d0630c4d91be9b17973fcb0230" member_type: VOTER last_known_addr { host: "127.6.235.67" port: 41399 } } peers { permanent_uuid: "37070660923a4173b9bfe2c5b3abc0c4" member_type: VOTER last_known_addr { host: "127.6.235.65" port: 37261 } } peers { permanent_uuid: "f820529e7c8e4f2f8341f373b1def2d0" member_type: VOTER last_known_addr { host: "127.6.235.66" port: 36697 } }
I20260928 02:14:15.249694 21528 catalog_manager.cc:5495] T b6e74e850a0e463f8677dd6a1e1982c4 P 37070660923a4173b9bfe2c5b3abc0c4 reported cstate change: term changed from 0 to 1, leader changed from <none> to 37070660923a4173b9bfe2c5b3abc0c4 (127.6.235.65). New cstate: current_term: 1 leader_uuid: "37070660923a4173b9bfe2c5b3abc0c4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f11b50d0630c4d91be9b17973fcb0230" member_type: VOTER last_known_addr { host: "127.6.235.67" port: 41399 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "37070660923a4173b9bfe2c5b3abc0c4" member_type: VOTER last_known_addr { host: "127.6.235.65" port: 37261 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f820529e7c8e4f2f8341f373b1def2d0" member_type: VOTER last_known_addr { host: "127.6.235.66" port: 36697 } health_report { overall_health: UNKNOWN } } }
I20260928 02:14:15.274470 21992 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:14:15.269160 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47566 (local address 127.6.235.65:37261)
0928 02:14:15.269353 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0928 02:14:15.269357 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:14:15.269374 (+ 17us) server_negotiation.cc:407] Connection header received
0928 02:14:15.269560 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:14:15.269564 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:14:15.269643 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:14:15.269770 (+ 127us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 02:14:15.270215 (+ 445us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:14:15.270946 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:14:15.272358 (+ 1412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:14:15.272771 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:14:15.272863 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:14:15.273448 (+ 585us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 02:14:15.273983 (+ 535us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 02:14:15.274094 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0928 02:14:15.274193 (+ 99us) server_negotiation.cc:299] Negotiation successful
0928 02:14:15.274260 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260928 02:14:15.283020 21932 raft_consensus.cc:1261] T b6e74e850a0e463f8677dd6a1e1982c4 P f11b50d0630c4d91be9b17973fcb0230 [term 1 FOLLOWER]: Refusing update from remote peer 37070660923a4173b9bfe2c5b3abc0c4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 02:14:15.283020 21801 raft_consensus.cc:1261] T b6e74e850a0e463f8677dd6a1e1982c4 P f820529e7c8e4f2f8341f373b1def2d0 [term 1 FOLLOWER]: Refusing update from remote peer 37070660923a4173b9bfe2c5b3abc0c4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 02:14:15.284003 22002 consensus_queue.cc:1035] T b6e74e850a0e463f8677dd6a1e1982c4 P 37070660923a4173b9bfe2c5b3abc0c4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f820529e7c8e4f2f8341f373b1def2d0" member_type: VOTER last_known_addr { host: "127.6.235.66" port: 36697 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 02:14:15.284368 22000 consensus_queue.cc:1035] T b6e74e850a0e463f8677dd6a1e1982c4 P 37070660923a4173b9bfe2c5b3abc0c4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f11b50d0630c4d91be9b17973fcb0230" member_type: VOTER last_known_addr { host: "127.6.235.67" port: 41399 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 02:14:15.296481 22007 mvcc.cc:205] Tried to move back new op lower bound from 7334140540028833792 to 7334140539888717824. Current Snapshot: MvccSnapshot[applied={T|T < 7334140540028833792}]
I20260928 02:14:15.314024 21528 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin'} at 127.0.0.1:35034:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 02:14:15.314320 21528 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin'} at 127.0.0.1:35034:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 02:14:15.318279 21528 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 65164f35d42e4ab5a76865f2884ee008: Sending DeleteTablet for 3 replicas of tablet b6e74e850a0e463f8677dd6a1e1982c4
I20260928 02:14:15.319538 21912 tablet_service.cc:1514] Processing DeleteTablet for tablet b6e74e850a0e463f8677dd6a1e1982c4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:14:15 UTC) from {username='kudu', principal='kudu/127.6.235.126@KRBTEST.COM'} at 127.0.0.1:58866
I20260928 02:14:15.319957 7085 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskTlaUuc/build/asan/bin/kudu pid 21566
I20260928 02:14:15.320011 21781 tablet_service.cc:1514] Processing DeleteTablet for tablet b6e74e850a0e463f8677dd6a1e1982c4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:14:15 UTC) from {username='kudu', principal='kudu/127.6.235.126@KRBTEST.COM'} at 127.0.0.1:48334
I20260928 02:14:15.320829 21649 tablet_service.cc:1514] Processing DeleteTablet for tablet b6e74e850a0e463f8677dd6a1e1982c4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:14:15 UTC) from {username='kudu', principal='kudu/127.6.235.126@KRBTEST.COM'} at 127.0.0.1:47550
I20260928 02:14:15.330670 22018 tablet_replica.cc:330] T b6e74e850a0e463f8677dd6a1e1982c4 P f820529e7c8e4f2f8341f373b1def2d0: stopping tablet replica
I20260928 02:14:15.332085 22018 raft_consensus.cc:2229] T b6e74e850a0e463f8677dd6a1e1982c4 P f820529e7c8e4f2f8341f373b1def2d0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 02:14:15.333572 22018 raft_consensus.cc:2258] T b6e74e850a0e463f8677dd6a1e1982c4 P f820529e7c8e4f2f8341f373b1def2d0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 02:14:15.337711 22018 ts_tablet_manager.cc:1905] T b6e74e850a0e463f8677dd6a1e1982c4 P f820529e7c8e4f2f8341f373b1def2d0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 02:14:15.340763 21992 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:14:15.321426 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47568 (local address 127.6.235.65:37261)
0928 02:14:15.321597 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0928 02:14:15.321603 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 02:14:15.321620 (+ 17us) server_negotiation.cc:407] Connection header received
0928 02:14:15.321673 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:14:15.321678 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:14:15.321753 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:14:15.321848 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:14:15.323835 (+ 1987us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:14:15.325177 (+ 1342us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:14:15.327131 (+ 1954us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:14:15.327844 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:14:15.327978 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:14:15.331797 (+ 3819us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:14:15.331812 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:14:15.331830 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:14:15.331888 (+ 58us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:14:15.336299 (+ 4411us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:14:15.337310 (+ 1011us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:14:15.337320 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:14:15.337323 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:14:15.337510 (+ 187us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:14:15.337987 (+ 477us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:14:15.337992 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:14:15.337996 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:14:15.338457 (+ 461us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:14:15.338762 (+ 305us) server_negotiation.cc:1035] Waiting for connection context
0928 02:14:15.340124 (+ 1362us) server_negotiation.cc:299] Negotiation successful
0928 02:14:15.340370 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260928 02:14:15.341646 21709 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 02:14:15.342093 22017 tablet_replica.cc:330] T b6e74e850a0e463f8677dd6a1e1982c4 P f11b50d0630c4d91be9b17973fcb0230: stopping tablet replica
I20260928 02:14:15.342582 22017 raft_consensus.cc:2229] T b6e74e850a0e463f8677dd6a1e1982c4 P f11b50d0630c4d91be9b17973fcb0230 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 02:14:15.343134 22017 raft_consensus.cc:2258] T b6e74e850a0e463f8677dd6a1e1982c4 P f11b50d0630c4d91be9b17973fcb0230 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 02:14:15.345568 22017 ts_tablet_manager.cc:1905] T b6e74e850a0e463f8677dd6a1e1982c4 P f11b50d0630c4d91be9b17973fcb0230: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 02:14:15.348526 22018 ts_tablet_manager.cc:1918] T b6e74e850a0e463f8677dd6a1e1982c4 P f820529e7c8e4f2f8341f373b1def2d0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 02:14:15.348701 22018 log.cc:1196] T b6e74e850a0e463f8677dd6a1e1982c4 P f820529e7c8e4f2f8341f373b1def2d0: Deleting WAL directory at /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790561432251112-7085-0/minicluster-data/ts-1/wal/wals/b6e74e850a0e463f8677dd6a1e1982c4
I20260928 02:14:15.349282 22018 ts_tablet_manager.cc:1939] T b6e74e850a0e463f8677dd6a1e1982c4 P f820529e7c8e4f2f8341f373b1def2d0: Deleting consensus metadata
I20260928 02:14:15.351424 21514 catalog_manager.cc:4841] TS f820529e7c8e4f2f8341f373b1def2d0 (127.6.235.66:36697): tablet b6e74e850a0e463f8677dd6a1e1982c4 (table test-table [id=7e8ba4d0c122435098a76188625b2db9]) successfully deleted
I20260928 02:14:15.351994 22017 ts_tablet_manager.cc:1918] T b6e74e850a0e463f8677dd6a1e1982c4 P f11b50d0630c4d91be9b17973fcb0230: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 02:14:15.352161 22017 log.cc:1196] T b6e74e850a0e463f8677dd6a1e1982c4 P f11b50d0630c4d91be9b17973fcb0230: Deleting WAL directory at /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790561432251112-7085-0/minicluster-data/ts-2/wal/wals/b6e74e850a0e463f8677dd6a1e1982c4
I20260928 02:14:15.352638 22017 ts_tablet_manager.cc:1939] T b6e74e850a0e463f8677dd6a1e1982c4 P f11b50d0630c4d91be9b17973fcb0230: Deleting consensus metadata
I20260928 02:14:15.354241 21513 catalog_manager.cc:4841] TS f11b50d0630c4d91be9b17973fcb0230 (127.6.235.67:41399): tablet b6e74e850a0e463f8677dd6a1e1982c4 (table test-table [id=7e8ba4d0c122435098a76188625b2db9]) successfully deleted
I20260928 02:14:15.443123 22019 tablet_replica.cc:330] T b6e74e850a0e463f8677dd6a1e1982c4 P 37070660923a4173b9bfe2c5b3abc0c4: stopping tablet replica
I20260928 02:14:15.443121 7085 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTlaUuc/build/asan/bin/kudu with pid 21566
I20260928 02:14:15.462092 7085 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskTlaUuc/build/asan/bin/kudu pid 21718
W20260928 02:14:15.462170 21513 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.6.235.65:37261 (error 108)
I20260928 02:14:15.464179 21991 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:14:15.463672 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.235.65:37261 (local address 127.0.0.1:47580)
0928 02:14:15.463867 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0928 02:14:15.463990 (+ 123us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.235.65:37261: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":72}
W20260928 02:14:15.464475 21513 catalog_manager.cc:4584] TS 37070660923a4173b9bfe2c5b3abc0c4 (127.6.235.65:37261): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet b6e74e850a0e463f8677dd6a1e1982c4: Network error: Client connection negotiation failed: client connection to 127.6.235.65:37261: connect: Connection refused (error 111)
I20260928 02:14:15.477221 21990 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:14:15.462915 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48348 (local address 127.6.235.66:36697)
0928 02:14:15.463379 (+ 464us) server_negotiation.cc:206] Beginning negotiation
0928 02:14:15.463388 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 02:14:15.463421 (+ 33us) server_negotiation.cc:407] Connection header received
0928 02:14:15.463495 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:14:15.463503 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:14:15.463596 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:14:15.463703 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:14:15.465373 (+ 1670us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:14:15.466137 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:14:15.467591 (+ 1454us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:14:15.468005 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:14:15.468110 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:14:15.471158 (+ 3048us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:14:15.471174 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:14:15.471190 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:14:15.471229 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:14:15.474327 (+ 3098us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:14:15.475104 (+ 777us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:14:15.475112 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:14:15.475114 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:14:15.475255 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:14:15.475601 (+ 346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:14:15.475604 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:14:15.475606 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:14:15.476016 (+ 410us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:14:15.476353 (+ 337us) server_negotiation.cc:1035] Waiting for connection context
0928 02:14:15.476647 (+ 294us) server_negotiation.cc:299] Negotiation successful
0928 02:14:15.476822 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":335}
I20260928 02:14:15.478066 21841 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 02:14:15.518421 21991 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:14:15.517822 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.235.65:37261 (local address 127.0.0.1:47592)
0928 02:14:15.518084 (+ 262us) negotiation.cc:107] Waiting for socket to connect
0928 02:14:15.518220 (+ 136us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.235.65:37261: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":108}
I20260928 02:14:15.519516 21991 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:14:15.519115 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.235.65:37261 (local address 127.0.0.1:47594)
0928 02:14:15.519273 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0928 02:14:15.519374 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.235.65:37261: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
I20260928 02:14:15.567536 7085 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTlaUuc/build/asan/bin/kudu with pid 21718
I20260928 02:14:15.579870 21991 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:14:15.579397 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.235.65:37261 (local address 127.0.0.1:47600)
0928 02:14:15.579614 (+ 217us) negotiation.cc:107] Waiting for socket to connect
0928 02:14:15.579698 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.235.65:37261: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":71}
I20260928 02:14:15.581032 21991 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:14:15.580622 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.235.65:37261 (local address 127.0.0.1:47610)
0928 02:14:15.580776 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0928 02:14:15.580850 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.235.65:37261: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
I20260928 02:14:15.589488 7085 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskTlaUuc/build/asan/bin/kudu pid 21849
I20260928 02:14:15.605912 21988 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:14:15.590306 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58872 (local address 127.6.235.67:41399)
0928 02:14:15.590522 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0928 02:14:15.590529 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 02:14:15.590573 (+ 44us) server_negotiation.cc:407] Connection header received
0928 02:14:15.590777 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:14:15.590783 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:14:15.590888 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:14:15.591013 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:14:15.592424 (+ 1411us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:14:15.593580 (+ 1156us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:14:15.594904 (+ 1324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:14:15.595590 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:14:15.595713 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:14:15.598440 (+ 2727us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:14:15.598459 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:14:15.598482 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:14:15.598523 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:14:15.602755 (+ 4232us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:14:15.603490 (+ 735us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:14:15.603501 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:14:15.603506 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:14:15.603670 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:14:15.604163 (+ 493us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:14:15.604176 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:14:15.604179 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:14:15.604580 (+ 401us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:14:15.604832 (+ 252us) server_negotiation.cc:1035] Waiting for connection context
0928 02:14:15.605252 (+ 420us) server_negotiation.cc:299] Negotiation successful
0928 02:14:15.605507 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260928 02:14:15.606576 21972 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 02:14:15.662063 21991 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:14:15.661435 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.235.65:37261 (local address 127.0.0.1:47618)
0928 02:14:15.661692 (+ 257us) negotiation.cc:107] Waiting for socket to connect
0928 02:14:15.661822 (+ 130us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.235.65:37261: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":92}
I20260928 02:14:15.663264 21991 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:14:15.662801 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.235.65:37261 (local address 127.0.0.1:47626)
0928 02:14:15.663027 (+ 226us) negotiation.cc:107] Waiting for socket to connect
0928 02:14:15.663120 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.235.65:37261: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":106}
I20260928 02:14:15.698589 7085 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTlaUuc/build/asan/bin/kudu with pid 21849
I20260928 02:14:15.717190 7085 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskTlaUuc/build/asan/bin/kudu pid 21496
I20260928 02:14:15.718537 21559 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 02:14:15.836338 7085 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTlaUuc/build/asan/bin/kudu with pid 21496
I20260928 02:14:15.836436 21991 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:14:15.835881 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.235.65:37261 (local address 127.0.0.1:47634)
0928 02:14:15.836125 (+ 244us) negotiation.cc:107] Waiting for socket to connect
0928 02:14:15.836243 (+ 118us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.235.65:37261: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":78}
I20260928 02:14:15.859747 7085 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTlaUuc/build/asan/bin/kudu
/tmp/dist-test-taskTlaUuc/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790561432251112-7085-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790561432251112-7085-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790561432251112-7085-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790561432251112-7085-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.235.126:40053
--webserver_interface=127.6.235.126
--webserver_port=35131
--keytab_file=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790561432251112-7085-0/krb5kdc/kudu_127.6.235.126.keytab
--principal=kudu/127.6.235.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.235.84:34573
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.235.126:40053
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790561432251112-7085-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790561432251112-7085-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790561432251112-7085-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:14:16.202306 22025 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:14:16.202615 22025 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:14:16.202791 22025 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:14:16.210093 22025 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 02:14:16.210201 22025 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:14:16.210287 22025 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:14:16.210371 22025 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 02:14:16.210438 22025 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 02:14:16.220445 22025 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.235.84:34573
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790561432251112-7085-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790561432251112-7085-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.235.126:40053
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790561432251112-7085-0/krb5kdc/kudu_127.6.235.126.keytab
--principal=kudu/127.6.235.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.235.126:40053
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790561432251112-7085-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.235.126
--webserver_port=35131
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790561432251112-7085-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 00:51:59 UTC on ccycloud.quasar-axgkwn.root.comops.site
build id 4676464
ASAN enabled
I20260928 02:14:16.221925 22025 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:14:16.223268 22025 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:14:16.235785 22031 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:14:16.235759 22033 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:14:16.236186 22030 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:14:16.236876 22025 server_base.cc:1034] running on GCE node
I20260928 02:14:16.238059 22025 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:14:16.239868 22025 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:14:16.241099 22025 hybrid_clock.cc:648] HybridClock initialized: now 1790561656241028 us; error 69 us; skew 500 ppm
I20260928 02:14:16.241475 22025 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 02:14:16 dist-test-slave-86gv krb5kdc[20943](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561656, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.235.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:14:16.244874 22025 init.cc:377] Logged in from keytab as kudu/127.6.235.126@KRBTEST.COM (short username kudu)
I20260928 02:14:16.246621 22025 webserver.cc:460] Webserver started at http://127.6.235.126:35131/ using document root <none> and password file <none>
I20260928 02:14:16.247354 22025 fs_manager.cc:359] Metadata directory not provided
I20260928 02:14:16.247512 22025 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:14:16.253437 22025 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.002s sys 0.001s
I20260928 02:14:16.256554 22040 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:14:16.258129 22025 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 02:14:16.258338 22025 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790561432251112-7085-0/minicluster-data/master-0/data,/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790561432251112-7085-0/minicluster-data/master-0/wal
uuid: "65164f35d42e4ab5a76865f2884ee008"
format_stamp: "Formatted at 2026-09-28 02:14:10 on dist-test-slave-86gv"
server_key: "004236509a5ac5a25bf575733d3d9620"
server_key_iv: "116136b2a19c5a524b3a72651389aeb2"
server_key_version: "encryptionkey@0"
I20260928 02:14:16.259406 22025 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790561432251112-7085-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790561432251112-7085-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790561432251112-7085-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:14:16.279649 22025 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:14:16.284035 22025 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:14:16.284329 22025 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:14:16.304147 22025 rpc_server.cc:307] RPC server started. Bound to: 127.6.235.126:40053
I20260928 02:14:16.304127 22091 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.235.126:40053 every 8 connection(s)
I20260928 02:14:16.305871 22025 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790561432251112-7085-0/minicluster-data/master-0/data/info.pb
I20260928 02:14:16.313300 22092 sys_catalog.cc:263] Verifying existing consensus state
I20260928 02:14:16.314666 7085 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTlaUuc/build/asan/bin/kudu as pid 22025
I20260928 02:14:16.315506 7085 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTlaUuc/build/asan/bin/kudu
/tmp/dist-test-taskTlaUuc/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790561432251112-7085-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790561432251112-7085-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790561432251112-7085-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790561432251112-7085-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.235.65:37261
--local_ip_for_outbound_sockets=127.6.235.65
--tserver_master_addrs=127.6.235.126:40053
--webserver_port=41415
--webserver_interface=127.6.235.65
--keytab_file=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790561432251112-7085-0/krb5kdc/kudu_127.6.235.65.keytab
--principal=kudu/127.6.235.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.235.84:34573
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790561432251112-7085-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790561432251112-7085-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790561432251112-7085-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 02:14:16.316833 22092 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 65164f35d42e4ab5a76865f2884ee008: Bootstrap starting.
I20260928 02:14:16.337535 22092 log.cc:824] T 00000000000000000000000000000000 P 65164f35d42e4ab5a76865f2884ee008: Log is configured to *not* fsync() on all Append() calls
I20260928 02:14:16.354334 22092 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 65164f35d42e4ab5a76865f2884ee008: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=7 ignored=0} mutations{seen=9 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 02:14:16.354943 22092 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 65164f35d42e4ab5a76865f2884ee008: Bootstrap complete.
I20260928 02:14:16.364472 22092 raft_consensus.cc:348] T 00000000000000000000000000000000 P 65164f35d42e4ab5a76865f2884ee008 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "65164f35d42e4ab5a76865f2884ee008" member_type: VOTER last_known_addr { host: "127.6.235.126" port: 40053 } }
I20260928 02:14:16.365514 22092 raft_consensus.cc:729] T 00000000000000000000000000000000 P 65164f35d42e4ab5a76865f2884ee008 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 65164f35d42e4ab5a76865f2884ee008, State: Initialized, Role: FOLLOWER
I20260928 02:14:16.366144 22092 consensus_queue.cc:260] T 00000000000000000000000000000000 P 65164f35d42e4ab5a76865f2884ee008 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "65164f35d42e4ab5a76865f2884ee008" member_type: VOTER last_known_addr { host: "127.6.235.126" port: 40053 } }
I20260928 02:14:16.366405 22092 raft_consensus.cc:388] T 00000000000000000000000000000000 P 65164f35d42e4ab5a76865f2884ee008 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 02:14:16.366539 22092 raft_consensus.cc:482] T 00000000000000000000000000000000 P 65164f35d42e4ab5a76865f2884ee008 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 02:14:16.366724 22092 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 65164f35d42e4ab5a76865f2884ee008 [term 2 FOLLOWER]: Advancing to term 3
I20260928 02:14:16.371183 22092 raft_consensus.cc:504] T 00000000000000000000000000000000 P 65164f35d42e4ab5a76865f2884ee008 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "65164f35d42e4ab5a76865f2884ee008" member_type: VOTER last_known_addr { host: "127.6.235.126" port: 40053 } }
I20260928 02:14:16.371829 22092 leader_election.cc:302] T 00000000000000000000000000000000 P 65164f35d42e4ab5a76865f2884ee008 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 65164f35d42e4ab5a76865f2884ee008; no voters:
I20260928 02:14:16.372542 22092 leader_election.cc:288] T 00000000000000000000000000000000 P 65164f35d42e4ab5a76865f2884ee008 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260928 02:14:16.372684 22096 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 65164f35d42e4ab5a76865f2884ee008 [term 3 FOLLOWER]: Leader election won for term 3
I20260928 02:14:16.373934 22096 raft_consensus.cc:686] T 00000000000000000000000000000000 P 65164f35d42e4ab5a76865f2884ee008 [term 3 LEADER]: Becoming Leader. State: Replica: 65164f35d42e4ab5a76865f2884ee008, State: Running, Role: LEADER
I20260928 02:14:16.374559 22096 consensus_queue.cc:237] T 00000000000000000000000000000000 P 65164f35d42e4ab5a76865f2884ee008 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "65164f35d42e4ab5a76865f2884ee008" member_type: VOTER last_known_addr { host: "127.6.235.126" port: 40053 } }
I20260928 02:14:16.374949 22092 sys_catalog.cc:564] T 00000000000000000000000000000000 P 65164f35d42e4ab5a76865f2884ee008 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 02:14:16.377173 22098 sys_catalog.cc:455] T 00000000000000000000000000000000 P 65164f35d42e4ab5a76865f2884ee008 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 65164f35d42e4ab5a76865f2884ee008. Latest consensus state: current_term: 3 leader_uuid: "65164f35d42e4ab5a76865f2884ee008" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "65164f35d42e4ab5a76865f2884ee008" member_type: VOTER last_known_addr { host: "127.6.235.126" port: 40053 } } }
I20260928 02:14:16.377774 22098 sys_catalog.cc:458] T 00000000000000000000000000000000 P 65164f35d42e4ab5a76865f2884ee008 [sys.catalog]: This master's current role is: LEADER
I20260928 02:14:16.378391 22097 sys_catalog.cc:455] T 00000000000000000000000000000000 P 65164f35d42e4ab5a76865f2884ee008 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "65164f35d42e4ab5a76865f2884ee008" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "65164f35d42e4ab5a76865f2884ee008" member_type: VOTER last_known_addr { host: "127.6.235.126" port: 40053 } } }
I20260928 02:14:16.378707 22097 sys_catalog.cc:458] T 00000000000000000000000000000000 P 65164f35d42e4ab5a76865f2884ee008 [sys.catalog]: This master's current role is: LEADER
I20260928 02:14:16.382999 22103 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 02:14:16.394495 22103 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 02:14:16.396605 22103 catalog_manager.cc:1259] Loaded cluster ID: 6dfacd5e0cbd4b6fb1ce99f299c840f2
I20260928 02:14:16.396752 22103 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
F20260928 02:14:16.404070 22103 catalog_manager.cc:1466] Initializing Kudu internal certificate authority failed: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7fabf7cad9ad google::LogMessage::Fail() at ??:0
I20260928 02:14:16.408263 22113 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
@ 0x7fabf7cb088a google::LogMessage::SendToLog() at ??:0
@ 0x7fabf7cad4de google::LogMessage::Flush() at ??:0
@ 0x7fabf7caea19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7fac0d6e4512 kudu::master::CatalogManager::PrepareForLeadershipTask()::$_6::operator()() at ??:0
@ 0x7fac0d6e1976 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7fac0d769610 kudu::master::CatalogManager::ElectedAsLeaderCb()::$_2::operator()() at ??:0
@ 0x7fac0d769439 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7fabfa8f9345 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7fabfa8fc91c kudu::ThreadPool::CreateThread()::$_2::operator()() at ??:0
@ 0x7fabfa8fc799 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7fabfa8d7eaf kudu::Thread::SuperviseThread() at ??:0
@ 0x7fabfdbb16db start_thread at ??:0
@ 0x7fabf6b4761f clone at ??:0
W20260928 02:14:16.716882 22094 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:14:16.717375 22094 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:14:16.717631 22094 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:14:16.726154 22094 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:14:16.726361 22094 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:14:16.726565 22094 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.235.65
I20260928 02:14:16.737859 22094 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.235.84:34573
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790561432251112-7085-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790561432251112-7085-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790561432251112-7085-0/krb5kdc/kudu_127.6.235.65.keytab
--principal=kudu/127.6.235.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.235.65:37261
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790561432251112-7085-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.6.235.65
--webserver_port=41415
--tserver_master_addrs=127.6.235.126:40053
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.235.65
--log_dir=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790561432251112-7085-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 00:51:59 UTC on ccycloud.quasar-axgkwn.root.comops.site
build id 4676464
ASAN enabled
I20260928 02:14:16.739368 22094 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:14:16.740816 22094 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:14:16.754973 22134 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:14:16.754987 22136 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:14:16.755797 22133 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:14:16.757908 22094 server_base.cc:1034] running on GCE node
I20260928 02:14:16.758867 22094 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:14:16.760285 22094 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:14:16.761585 22094 hybrid_clock.cc:648] HybridClock initialized: now 1790561656761542 us; error 40 us; skew 500 ppm
I20260928 02:14:16.761894 22094 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 02:14:16 dist-test-slave-86gv krb5kdc[20943](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561656, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.235.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:14:16.771485 22094 init.cc:377] Logged in from keytab as kudu/127.6.235.65@KRBTEST.COM (short username kudu)
I20260928 02:14:16.773298 22094 webserver.cc:460] Webserver started at http://127.6.235.65:41415/ using document root <none> and password file <none>
I20260928 02:14:16.774107 22094 fs_manager.cc:359] Metadata directory not provided
I20260928 02:14:16.774243 22094 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:14:16.780121 22094 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.002s sys 0.004s
I20260928 02:14:16.783486 22143 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:14:16.785166 22094 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260928 02:14:16.785364 22094 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790561432251112-7085-0/minicluster-data/ts-0/data,/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790561432251112-7085-0/minicluster-data/ts-0/wal
uuid: "37070660923a4173b9bfe2c5b3abc0c4"
format_stamp: "Formatted at 2026-09-28 02:14:10 on dist-test-slave-86gv"
server_key: "1b06c11186b1b991199074c31887352b"
server_key_iv: "1f600e3577ba6b34682b7fb5036eb7df"
server_key_version: "encryptionkey@0"
I20260928 02:14:16.786434 22094 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790561432251112-7085-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790561432251112-7085-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790561432251112-7085-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:14:16.801196 22094 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:14:16.806092 22094 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:14:16.806445 22094 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:14:16.807593 22094 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:14:16.809650 22150 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
I20260928 02:14:16.816612 22094 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260928 02:14:16.816758 22094 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.007s user 0.001s sys 0.000s
I20260928 02:14:16.816941 22094 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260928 02:14:16.820307 22094 ts_tablet_manager.cc:610] Registered 1 tablets
I20260928 02:14:16.820399 22094 ts_tablet_manager.cc:589] Time spent register tablets: real 0.003s user 0.000s sys 0.000s
I20260928 02:14:16.821030 22150 tablet_bootstrap.cc:492] T b6e74e850a0e463f8677dd6a1e1982c4 P 37070660923a4173b9bfe2c5b3abc0c4: Bootstrap starting.
I20260928 02:14:16.854771 22150 log.cc:824] T b6e74e850a0e463f8677dd6a1e1982c4 P 37070660923a4173b9bfe2c5b3abc0c4: Log is configured to *not* fsync() on all Append() calls
I20260928 02:14:16.860265 22094 rpc_server.cc:307] RPC server started. Bound to: 127.6.235.65:37261
I20260928 02:14:16.860272 22257 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.235.65:37261 every 8 connection(s)
I20260928 02:14:16.861883 22094 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790561432251112-7085-0/minicluster-data/ts-0/data/info.pb
I20260928 02:14:16.867202 22260 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:14:16.865063 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.235.126:40053 (local address 127.6.235.65:37669)
0928 02:14:16.866010 (+ 947us) negotiation.cc:107] Waiting for socket to connect
0928 02:14:16.866265 (+ 255us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.235.126:40053: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":689,"thread_start_us":436,"threads_started":1}
W20260928 02:14:16.867856 22258 heartbeater.cc:644] Failed to heartbeat to 127.6.235.126:40053 (0 consecutive failures): Network error: Failed to ping master at 127.6.235.126:40053: Client connection negotiation failed: client connection to 127.6.235.126:40053: connect: Connection refused (error 111)
I20260928 02:14:16.868768 22260 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:14:16.868436 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.235.126:40053 (local address 127.6.235.65:35489)
0928 02:14:16.868591 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0928 02:14:16.868666 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.235.126:40053: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
I20260928 02:14:16.869567 22150 tablet_bootstrap.cc:492] T b6e74e850a0e463f8677dd6a1e1982c4 P 37070660923a4173b9bfe2c5b3abc0c4: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 02:14:16.869748 22260 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:14:16.869327 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.235.126:40053 (local address 127.6.235.65:52845)
0928 02:14:16.869488 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0928 02:14:16.869593 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.235.126:40053: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
W20260928 02:14:16.870042 22258 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260928 02:14:16.870225 22150 tablet_bootstrap.cc:492] T b6e74e850a0e463f8677dd6a1e1982c4 P 37070660923a4173b9bfe2c5b3abc0c4: Bootstrap complete.
I20260928 02:14:16.871420 7085 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTlaUuc/build/asan/bin/kudu as pid 22094
I20260928 02:14:16.871438 22150 ts_tablet_manager.cc:1397] T b6e74e850a0e463f8677dd6a1e1982c4 P 37070660923a4173b9bfe2c5b3abc0c4: Time spent bootstrapping tablet: real 0.051s user 0.023s sys 0.024s
I20260928 02:14:16.872083 7085 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTlaUuc/build/asan/bin/kudu
/tmp/dist-test-taskTlaUuc/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790561432251112-7085-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790561432251112-7085-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790561432251112-7085-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790561432251112-7085-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.235.66:36697
--local_ip_for_outbound_sockets=127.6.235.66
--tserver_master_addrs=127.6.235.126:40053
--webserver_port=45689
--webserver_interface=127.6.235.66
--keytab_file=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790561432251112-7085-0/krb5kdc/kudu_127.6.235.66.keytab
--principal=kudu/127.6.235.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.235.84:34573
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790561432251112-7085-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790561432251112-7085-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790561432251112-7085-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 02:14:16.877905 22150 raft_consensus.cc:348] T b6e74e850a0e463f8677dd6a1e1982c4 P 37070660923a4173b9bfe2c5b3abc0c4 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f11b50d0630c4d91be9b17973fcb0230" member_type: VOTER last_known_addr { host: "127.6.235.67" port: 41399 } } peers { permanent_uuid: "37070660923a4173b9bfe2c5b3abc0c4" member_type: VOTER last_known_addr { host: "127.6.235.65" port: 37261 } } peers { permanent_uuid: "f820529e7c8e4f2f8341f373b1def2d0" member_type: VOTER last_known_addr { host: "127.6.235.66" port: 36697 } }
I20260928 02:14:16.878818 22150 raft_consensus.cc:729] T b6e74e850a0e463f8677dd6a1e1982c4 P 37070660923a4173b9bfe2c5b3abc0c4 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 37070660923a4173b9bfe2c5b3abc0c4, State: Initialized, Role: FOLLOWER
I20260928 02:14:16.879516 22150 consensus_queue.cc:260] T b6e74e850a0e463f8677dd6a1e1982c4 P 37070660923a4173b9bfe2c5b3abc0c4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f11b50d0630c4d91be9b17973fcb0230" member_type: VOTER last_known_addr { host: "127.6.235.67" port: 41399 } } peers { permanent_uuid: "37070660923a4173b9bfe2c5b3abc0c4" member_type: VOTER last_known_addr { host: "127.6.235.65" port: 37261 } } peers { permanent_uuid: "f820529e7c8e4f2f8341f373b1def2d0" member_type: VOTER last_known_addr { host: "127.6.235.66" port: 36697 } }
I20260928 02:14:16.881269 22150 ts_tablet_manager.cc:1428] T b6e74e850a0e463f8677dd6a1e1982c4 P 37070660923a4173b9bfe2c5b3abc0c4: Time spent starting tablet: real 0.010s user 0.011s sys 0.001s
I20260928 02:14:16.881520 22260 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:14:16.880961 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.235.126:40053 (local address 127.6.235.65:47067)
0928 02:14:16.881149 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0928 02:14:16.881247 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.235.126:40053: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
I20260928 02:14:16.882483 22260 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:14:16.882042 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.235.126:40053 (local address 127.6.235.65:46811)
0928 02:14:16.882230 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0928 02:14:16.882322 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.235.126:40053: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
W20260928 02:14:17.215090 22261 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:14:17.215480 22261 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:14:17.215580 22261 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:14:17.222949 22261 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:14:17.223071 22261 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:14:17.223186 22261 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.235.66
I20260928 02:14:17.233474 22261 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.235.84:34573
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790561432251112-7085-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790561432251112-7085-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790561432251112-7085-0/krb5kdc/kudu_127.6.235.66.keytab
--principal=kudu/127.6.235.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.235.66:36697
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790561432251112-7085-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.6.235.66
--webserver_port=45689
--tserver_master_addrs=127.6.235.126:40053
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.235.66
--log_dir=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790561432251112-7085-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 00:51:59 UTC on ccycloud.quasar-axgkwn.root.comops.site
build id 4676464
ASAN enabled
I20260928 02:14:17.234895 22261 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:14:17.236315 22261 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:14:17.250638 22267 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:14:17.250588 22270 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:14:17.250607 22268 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:14:17.251924 22261 server_base.cc:1034] running on GCE node
I20260928 02:14:17.252645 22261 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:14:17.253795 22261 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:14:17.255054 22261 hybrid_clock.cc:648] HybridClock initialized: now 1790561657255003 us; error 44 us; skew 500 ppm
I20260928 02:14:17.255445 22261 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 02:14:17 dist-test-slave-86gv krb5kdc[20943](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561657, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.235.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:14:17.259164 22261 init.cc:377] Logged in from keytab as kudu/127.6.235.66@KRBTEST.COM (short username kudu)
I20260928 02:14:17.260799 22261 webserver.cc:460] Webserver started at http://127.6.235.66:45689/ using document root <none> and password file <none>
I20260928 02:14:17.261626 22261 fs_manager.cc:359] Metadata directory not provided
I20260928 02:14:17.261785 22261 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:14:17.267586 22261 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20260928 02:14:17.270916 22277 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:14:17.272545 22261 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.001s
I20260928 02:14:17.272751 22261 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790561432251112-7085-0/minicluster-data/ts-1/data,/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790561432251112-7085-0/minicluster-data/ts-1/wal
uuid: "f820529e7c8e4f2f8341f373b1def2d0"
format_stamp: "Formatted at 2026-09-28 02:14:11 on dist-test-slave-86gv"
server_key: "2121432c3e1f641baf637d3823de8d3d"
server_key_iv: "659dd1f1aa6458b3c06d8512e968d2c9"
server_key_version: "encryptionkey@0"
I20260928 02:14:17.273854 22261 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790561432251112-7085-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790561432251112-7085-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790561432251112-7085-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:14:17.292665 22261 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:14:17.298015 22261 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:14:17.298394 22261 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:14:17.299606 22261 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:14:17.301259 22261 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:14:17.301411 22261 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:14:17.301579 22261 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:14:17.301687 22261 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:14:17.338822 22261 rpc_server.cc:307] RPC server started. Bound to: 127.6.235.66:36697
I20260928 02:14:17.338908 22389 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.235.66:36697 every 8 connection(s)
I20260928 02:14:17.340407 22261 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790561432251112-7085-0/minicluster-data/ts-1/data/info.pb
I20260928 02:14:17.345422 22392 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:14:17.343516 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.235.126:40053 (local address 127.6.235.66:54975)
0928 02:14:17.344218 (+ 702us) negotiation.cc:107] Waiting for socket to connect
0928 02:14:17.344459 (+ 241us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.235.126:40053: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":453,"thread_start_us":274,"threads_started":1}
W20260928 02:14:17.346298 22390 heartbeater.cc:644] Failed to heartbeat to 127.6.235.126:40053 (0 consecutive failures): Network error: Failed to ping master at 127.6.235.126:40053: Client connection negotiation failed: client connection to 127.6.235.126:40053: connect: Connection refused (error 111)
I20260928 02:14:17.347318 22392 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:14:17.346987 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.235.126:40053 (local address 127.6.235.66:50723)
0928 02:14:17.347145 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0928 02:14:17.347214 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.235.126:40053: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
I20260928 02:14:17.348425 7085 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTlaUuc/build/asan/bin/kudu as pid 22261
I20260928 02:14:17.348448 22392 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:14:17.347947 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.235.126:40053 (local address 127.6.235.66:54377)
0928 02:14:17.348163 (+ 216us) negotiation.cc:107] Waiting for socket to connect
0928 02:14:17.348268 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.235.126:40053: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":72}
W20260928 02:14:17.348737 22390 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260928 02:14:17.349356 7085 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTlaUuc/build/asan/bin/kudu
/tmp/dist-test-taskTlaUuc/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790561432251112-7085-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790561432251112-7085-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790561432251112-7085-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790561432251112-7085-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.235.67:41399
--local_ip_for_outbound_sockets=127.6.235.67
--tserver_master_addrs=127.6.235.126:40053
--webserver_port=40207
--webserver_interface=127.6.235.67
--keytab_file=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790561432251112-7085-0/krb5kdc/kudu_127.6.235.67.keytab
--principal=kudu/127.6.235.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.235.84:34573
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790561432251112-7085-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790561432251112-7085-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790561432251112-7085-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:14:17.686409 22393 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:14:17.686790 22393 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:14:17.686980 22393 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:14:17.695529 22393 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:14:17.695685 22393 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:14:17.695878 22393 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.235.67
I20260928 02:14:17.705822 22393 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.235.84:34573
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790561432251112-7085-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790561432251112-7085-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790561432251112-7085-0/krb5kdc/kudu_127.6.235.67.keytab
--principal=kudu/127.6.235.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.235.67:41399
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790561432251112-7085-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.6.235.67
--webserver_port=40207
--tserver_master_addrs=127.6.235.126:40053
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.235.67
--log_dir=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790561432251112-7085-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 00:51:59 UTC on ccycloud.quasar-axgkwn.root.comops.site
build id 4676464
ASAN enabled
I20260928 02:14:17.707350 22393 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:14:17.708768 22393 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:14:17.723145 22398 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:14:17.723232 22399 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:14:17.723188 22401 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:14:17.724596 22393 server_base.cc:1034] running on GCE node
I20260928 02:14:17.725275 22393 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:14:17.726222 22393 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:14:17.727429 22393 hybrid_clock.cc:648] HybridClock initialized: now 1790561657727362 us; error 57 us; skew 500 ppm
I20260928 02:14:17.727759 22393 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 02:14:17 dist-test-slave-86gv krb5kdc[20943](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561657, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.235.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:14:17.731019 22393 init.cc:377] Logged in from keytab as kudu/127.6.235.67@KRBTEST.COM (short username kudu)
I20260928 02:14:17.732666 22393 webserver.cc:460] Webserver started at http://127.6.235.67:40207/ using document root <none> and password file <none>
I20260928 02:14:17.733530 22393 fs_manager.cc:359] Metadata directory not provided
I20260928 02:14:17.733664 22393 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:14:17.739543 22393 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20260928 02:14:17.742772 22408 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:14:17.744263 22393 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260928 02:14:17.744465 22393 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790561432251112-7085-0/minicluster-data/ts-2/data,/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790561432251112-7085-0/minicluster-data/ts-2/wal
uuid: "f11b50d0630c4d91be9b17973fcb0230"
format_stamp: "Formatted at 2026-09-28 02:14:11 on dist-test-slave-86gv"
server_key: "51fe6d056371e4b2efa727b487ae6e62"
server_key_iv: "32e2a67cf03cccfcde49d183927eee55"
server_key_version: "encryptionkey@0"
I20260928 02:14:17.745652 22393 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790561432251112-7085-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790561432251112-7085-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790561432251112-7085-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:14:17.767693 22393 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:14:17.772667 22393 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:14:17.773034 22393 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:14:17.774187 22393 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:14:17.776144 22393 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:14:17.776319 22393 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:14:17.776495 22393 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:14:17.776605 22393 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:14:17.817368 22393 rpc_server.cc:307] RPC server started. Bound to: 127.6.235.67:41399
I20260928 02:14:17.817430 22520 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.235.67:41399 every 8 connection(s)
I20260928 02:14:17.819119 22393 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790561432251112-7085-0/minicluster-data/ts-2/data/info.pb
I20260928 02:14:17.823858 7085 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTlaUuc/build/asan/bin/kudu as pid 22393
I20260928 02:14:17.824420 22523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:14:17.822312 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.235.126:40053 (local address 127.6.235.67:34515)
0928 02:14:17.823108 (+ 796us) negotiation.cc:107] Waiting for socket to connect
0928 02:14:17.823359 (+ 251us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.235.126:40053: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":502,"thread_start_us":303,"threads_started":1}
W20260928 02:14:17.825263 22521 heartbeater.cc:644] Failed to heartbeat to 127.6.235.126:40053 (0 consecutive failures): Network error: Failed to ping master at 127.6.235.126:40053: Client connection negotiation failed: client connection to 127.6.235.126:40053: connect: Connection refused (error 111)
I20260928 02:14:17.826166 22523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:14:17.825839 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.235.126:40053 (local address 127.6.235.67:35977)
0928 02:14:17.825997 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0928 02:14:17.826063 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.235.126:40053: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":70}
I20260928 02:14:17.826996 22523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:14:17.826665 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.235.126:40053 (local address 127.6.235.67:41559)
0928 02:14:17.826781 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0928 02:14:17.826853 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.235.126:40053: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260928 02:14:17.827224 22521 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260928 02:14:17.884119 22525 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:14:17.883355 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.235.126:40053 (local address 127.6.235.65:45963)
0928 02:14:17.883814 (+ 459us) negotiation.cc:107] Waiting for socket to connect
0928 02:14:17.883910 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.235.126:40053: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":362,"thread_start_us":219,"threads_started":1}
I20260928 02:14:18.350322 22526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:14:18.349334 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.235.126:40053 (local address 127.6.235.66:50223)
0928 02:14:18.349938 (+ 604us) negotiation.cc:107] Waiting for socket to connect
0928 02:14:18.350067 (+ 129us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.235.126:40053: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":475,"thread_start_us":350,"threads_started":1}
I20260928 02:14:18.493052 22527 raft_consensus.cc:482] T b6e74e850a0e463f8677dd6a1e1982c4 P 37070660923a4173b9bfe2c5b3abc0c4 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 02:14:18.493350 22527 raft_consensus.cc:504] T b6e74e850a0e463f8677dd6a1e1982c4 P 37070660923a4173b9bfe2c5b3abc0c4 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f11b50d0630c4d91be9b17973fcb0230" member_type: VOTER last_known_addr { host: "127.6.235.67" port: 41399 } } peers { permanent_uuid: "37070660923a4173b9bfe2c5b3abc0c4" member_type: VOTER last_known_addr { host: "127.6.235.65" port: 37261 } } peers { permanent_uuid: "f820529e7c8e4f2f8341f373b1def2d0" member_type: VOTER last_known_addr { host: "127.6.235.66" port: 36697 } }
I20260928 02:14:18.495160 22527 leader_election.cc:288] T b6e74e850a0e463f8677dd6a1e1982c4 P 37070660923a4173b9bfe2c5b3abc0c4 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f11b50d0630c4d91be9b17973fcb0230 (127.6.235.67:41399), f820529e7c8e4f2f8341f373b1def2d0 (127.6.235.66:36697)
Sep 28 02:14:18 dist-test-slave-86gv krb5kdc[20943](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561656, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.235.65@KRBTEST.COM for kudu/127.6.235.67@KRBTEST.COM
Sep 28 02:14:18 dist-test-slave-86gv krb5kdc[20943](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561656, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.235.65@KRBTEST.COM for kudu/127.6.235.66@KRBTEST.COM
I20260928 02:14:18.536033 22530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:14:18.495779 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.235.65:39469 (local address 127.6.235.67:41399)
0928 02:14:18.507729 (+ 11950us) server_negotiation.cc:206] Beginning negotiation
0928 02:14:18.507755 (+ 26us) server_negotiation.cc:399] Waiting for connection header
0928 02:14:18.508760 (+ 1005us) server_negotiation.cc:407] Connection header received
0928 02:14:18.512377 (+ 3617us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:14:18.512401 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:14:18.513054 (+ 653us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:14:18.513494 (+ 440us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:14:18.516789 (+ 3295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:14:18.518324 (+ 1535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:14:18.521308 (+ 2984us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:14:18.522501 (+ 1193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:14:18.522986 (+ 485us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:14:18.528460 (+ 5474us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:14:18.528473 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:14:18.528495 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:14:18.528567 (+ 72us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:14:18.531824 (+ 3257us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:14:18.532861 (+ 1037us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:14:18.532869 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:14:18.532881 (+ 12us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:14:18.533116 (+ 235us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:14:18.533639 (+ 523us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:14:18.533646 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:14:18.533651 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:14:18.534233 (+ 582us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:14:18.534428 (+ 195us) server_negotiation.cc:1035] Waiting for connection context
0928 02:14:18.535209 (+ 781us) server_negotiation.cc:299] Negotiation successful
0928 02:14:18.535594 (+ 385us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11674,"thread_start_us":11517,"threads_started":1}
I20260928 02:14:18.536036 22531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:14:18.495836 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.235.65:39737 (local address 127.6.235.66:36697)
0928 02:14:18.508161 (+ 12325us) server_negotiation.cc:206] Beginning negotiation
0928 02:14:18.508180 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0928 02:14:18.508789 (+ 609us) server_negotiation.cc:407] Connection header received
0928 02:14:18.512480 (+ 3691us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:14:18.512489 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:14:18.513054 (+ 565us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:14:18.513494 (+ 440us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:14:18.516774 (+ 3280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:14:18.518234 (+ 1460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:14:18.521302 (+ 3068us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:14:18.522502 (+ 1200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:14:18.522899 (+ 397us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:14:18.528690 (+ 5791us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:14:18.528702 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:14:18.528721 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:14:18.528788 (+ 67us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:14:18.531957 (+ 3169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:14:18.532867 (+ 910us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:14:18.532874 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:14:18.532880 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:14:18.533114 (+ 234us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:14:18.533585 (+ 471us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:14:18.533592 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:14:18.533600 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:14:18.534203 (+ 603us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:14:18.534416 (+ 213us) server_negotiation.cc:1035] Waiting for connection context
0928 02:14:18.535233 (+ 817us) server_negotiation.cc:299] Negotiation successful
0928 02:14:18.535630 (+ 397us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12120,"thread_start_us":11935,"threads_started":1}
I20260928 02:14:18.536073 22528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:14:18.495568 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.235.66:36697 (local address 127.6.235.65:39737)
0928 02:14:18.508656 (+ 13088us) negotiation.cc:107] Waiting for socket to connect
0928 02:14:18.508676 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0928 02:14:18.509328 (+ 652us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:14:18.513796 (+ 4468us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:14:18.513801 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:14:18.515070 (+ 1269us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:14:18.516496 (+ 1426us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:14:18.516529 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:14:18.518492 (+ 1963us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:14:18.518500 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:14:18.521078 (+ 2578us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:14:18.521102 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:14:18.522830 (+ 1728us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:14:18.522845 (+ 15us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:14:18.523358 (+ 513us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:14:18.524585 (+ 1227us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:14:18.524628 (+ 43us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:14:18.528308 (+ 3680us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:14:18.532132 (+ 3824us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:14:18.532137 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:14:18.532141 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:14:18.532697 (+ 556us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:14:18.533242 (+ 545us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:14:18.533247 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:14:18.533250 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:14:18.533461 (+ 211us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:14:18.534360 (+ 899us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:14:18.534368 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:14:18.534707 (+ 339us) client_negotiation.cc:769] Sending connection context
0928 02:14:18.534982 (+ 275us) client_negotiation.cc:240] Negotiation successful
0928 02:14:18.535583 (+ 601us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12984,"mutex_wait_us":615,"spinlock_wait_cycles":117248,"thread_start_us":12584,"threads_started":1}
I20260928 02:14:18.536818 22529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:14:18.495535 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.235.67:41399 (local address 127.6.235.65:39469)
0928 02:14:18.508523 (+ 12988us) negotiation.cc:107] Waiting for socket to connect
0928 02:14:18.508573 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0928 02:14:18.509328 (+ 755us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:14:18.513750 (+ 4422us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:14:18.513764 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:14:18.515070 (+ 1306us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:14:18.516496 (+ 1426us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:14:18.516538 (+ 42us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:14:18.518515 (+ 1977us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:14:18.518519 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:14:18.521078 (+ 2559us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:14:18.521102 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:14:18.522862 (+ 1760us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:14:18.522868 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:14:18.523359 (+ 491us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:14:18.524585 (+ 1226us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:14:18.524628 (+ 43us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:14:18.528005 (+ 3377us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:14:18.532024 (+ 4019us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:14:18.532035 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:14:18.532052 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:14:18.532678 (+ 626us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:14:18.533249 (+ 571us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:14:18.533254 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:14:18.533257 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:14:18.533474 (+ 217us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:14:18.534371 (+ 897us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:14:18.534384 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:14:18.534707 (+ 323us) client_negotiation.cc:769] Sending connection context
0928 02:14:18.534983 (+ 276us) client_negotiation.cc:240] Negotiation successful
0928 02:14:18.535751 (+ 768us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12810,"mutex_wait_us":704,"spinlock_wait_cycles":43264,"thread_start_us":12801,"threads_started":1}
W20260928 02:14:18.539844 22145 leader_election.cc:341] T b6e74e850a0e463f8677dd6a1e1982c4 P 37070660923a4173b9bfe2c5b3abc0c4 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer f820529e7c8e4f2f8341f373b1def2d0 (127.6.235.66:36697): Not found: Tablet not found: b6e74e850a0e463f8677dd6a1e1982c4
W20260928 02:14:18.540174 22144 leader_election.cc:341] T b6e74e850a0e463f8677dd6a1e1982c4 P 37070660923a4173b9bfe2c5b3abc0c4 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer f11b50d0630c4d91be9b17973fcb0230 (127.6.235.67:41399): Not found: Tablet not found: b6e74e850a0e463f8677dd6a1e1982c4
I20260928 02:14:18.540395 22144 leader_election.cc:302] T b6e74e850a0e463f8677dd6a1e1982c4 P 37070660923a4173b9bfe2c5b3abc0c4 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 37070660923a4173b9bfe2c5b3abc0c4; no voters: f11b50d0630c4d91be9b17973fcb0230, f820529e7c8e4f2f8341f373b1def2d0
I20260928 02:14:18.540908 22527 raft_consensus.cc:2726] T b6e74e850a0e463f8677dd6a1e1982c4 P 37070660923a4173b9bfe2c5b3abc0c4 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260928 02:14:18.828701 22532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:14:18.827799 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.235.126:40053 (local address 127.6.235.67:57991)
0928 02:14:18.828292 (+ 493us) negotiation.cc:107] Waiting for socket to connect
0928 02:14:18.828461 (+ 169us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.235.126:40053: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":387,"thread_start_us":251,"threads_started":1}
I20260928 02:14:18.885586 22529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:14:18.885160 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.235.126:40053 (local address 127.6.235.65:36153)
0928 02:14:18.885333 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0928 02:14:18.885439 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.235.126:40053: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
I20260928 02:14:19.352056 22533 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:14:19.351270 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.235.126:40053 (local address 127.6.235.66:33025)
0928 02:14:19.351758 (+ 488us) negotiation.cc:107] Waiting for socket to connect
0928 02:14:19.351850 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.235.126:40053: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":382,"thread_start_us":269,"threads_started":1}
I20260928 02:14:19.830603 22534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:14:19.829812 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.235.126:40053 (local address 127.6.235.67:41553)
0928 02:14:19.830316 (+ 504us) negotiation.cc:107] Waiting for socket to connect
0928 02:14:19.830436 (+ 120us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.235.126:40053: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":371,"thread_start_us":253,"threads_started":1}
I20260928 02:14:19.887539 22535 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:14:19.886585 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.235.126:40053 (local address 127.6.235.65:57625)
0928 02:14:19.887147 (+ 562us) negotiation.cc:107] Waiting for socket to connect
0928 02:14:19.887288 (+ 141us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.235.126:40053: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":449,"thread_start_us":246,"threads_started":1}
I20260928 02:14:20.353900 22536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:14:20.353075 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.235.126:40053 (local address 127.6.235.66:41627)
0928 02:14:20.353623 (+ 548us) negotiation.cc:107] Waiting for socket to connect
0928 02:14:20.353714 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.235.126:40053: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":425,"thread_start_us":312,"threads_started":1}
I20260928 02:14:20.470350 22537 raft_consensus.cc:482] T b6e74e850a0e463f8677dd6a1e1982c4 P 37070660923a4173b9bfe2c5b3abc0c4 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 02:14:20.470568 22537 raft_consensus.cc:504] T b6e74e850a0e463f8677dd6a1e1982c4 P 37070660923a4173b9bfe2c5b3abc0c4 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f11b50d0630c4d91be9b17973fcb0230" member_type: VOTER last_known_addr { host: "127.6.235.67" port: 41399 } } peers { permanent_uuid: "37070660923a4173b9bfe2c5b3abc0c4" member_type: VOTER last_known_addr { host: "127.6.235.65" port: 37261 } } peers { permanent_uuid: "f820529e7c8e4f2f8341f373b1def2d0" member_type: VOTER last_known_addr { host: "127.6.235.66" port: 36697 } }
I20260928 02:14:20.471524 22537 leader_election.cc:288] T b6e74e850a0e463f8677dd6a1e1982c4 P 37070660923a4173b9bfe2c5b3abc0c4 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f11b50d0630c4d91be9b17973fcb0230 (127.6.235.67:41399), f820529e7c8e4f2f8341f373b1def2d0 (127.6.235.66:36697)
W20260928 02:14:20.472625 22144 leader_election.cc:341] T b6e74e850a0e463f8677dd6a1e1982c4 P 37070660923a4173b9bfe2c5b3abc0c4 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer f11b50d0630c4d91be9b17973fcb0230 (127.6.235.67:41399): Not found: Tablet not found: b6e74e850a0e463f8677dd6a1e1982c4
W20260928 02:14:20.472833 22145 leader_election.cc:341] T b6e74e850a0e463f8677dd6a1e1982c4 P 37070660923a4173b9bfe2c5b3abc0c4 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer f820529e7c8e4f2f8341f373b1def2d0 (127.6.235.66:36697): Not found: Tablet not found: b6e74e850a0e463f8677dd6a1e1982c4
I20260928 02:14:20.473050 22145 leader_election.cc:302] T b6e74e850a0e463f8677dd6a1e1982c4 P 37070660923a4173b9bfe2c5b3abc0c4 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 37070660923a4173b9bfe2c5b3abc0c4; no voters: f11b50d0630c4d91be9b17973fcb0230, f820529e7c8e4f2f8341f373b1def2d0
I20260928 02:14:20.473361 22537 raft_consensus.cc:2726] T b6e74e850a0e463f8677dd6a1e1982c4 P 37070660923a4173b9bfe2c5b3abc0c4 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260928 02:14:20.832392 22538 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:14:20.831675 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.235.126:40053 (local address 127.6.235.67:42737)
0928 02:14:20.832121 (+ 446us) negotiation.cc:107] Waiting for socket to connect
0928 02:14:20.832202 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.235.126:40053: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":338,"thread_start_us":235,"threads_started":1}
I20260928 02:14:20.889446 22539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:14:20.888576 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.235.126:40053 (local address 127.6.235.65:58447)
0928 02:14:20.889148 (+ 572us) negotiation.cc:107] Waiting for socket to connect
0928 02:14:20.889257 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.235.126:40053: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":464,"thread_start_us":238,"threads_started":1}
I20260928 02:14:21.355937 22540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:14:21.354898 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.235.126:40053 (local address 127.6.235.66:33301)
0928 02:14:21.355434 (+ 536us) negotiation.cc:107] Waiting for socket to connect
0928 02:14:21.355527 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.235.126:40053: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":417,"thread_start_us":281,"threads_started":1}
I20260928 02:14:21.834124 22541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:14:21.833421 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.235.126:40053 (local address 127.6.235.67:45301)
0928 02:14:21.833861 (+ 440us) negotiation.cc:107] Waiting for socket to connect
0928 02:14:21.833948 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.235.126:40053: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":339,"thread_start_us":225,"threads_started":1}
I20260928 02:14:21.891450 22542 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:14:21.890592 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.235.126:40053 (local address 127.6.235.65:59481)
0928 02:14:21.891162 (+ 570us) negotiation.cc:107] Waiting for socket to connect
0928 02:14:21.891252 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.235.126:40053: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":464,"thread_start_us":344,"threads_started":1}
I20260928 02:14:22.358000 22543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:14:22.357203 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.235.126:40053 (local address 127.6.235.66:55725)
0928 02:14:22.357656 (+ 453us) negotiation.cc:107] Waiting for socket to connect
0928 02:14:22.357822 (+ 166us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.235.126:40053: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":348,"thread_start_us":236,"threads_started":1}
I20260928 02:14:22.836084 22544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:14:22.835298 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.235.126:40053 (local address 127.6.235.67:34935)
0928 02:14:22.835793 (+ 495us) negotiation.cc:107] Waiting for socket to connect
0928 02:14:22.835916 (+ 123us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.235.126:40053: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":389,"thread_start_us":279,"threads_started":1}
I20260928 02:14:22.893445 22545 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:14:22.892417 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.235.126:40053 (local address 127.6.235.65:44381)
0928 02:14:22.893054 (+ 637us) negotiation.cc:107] Waiting for socket to connect
0928 02:14:22.893223 (+ 169us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.235.126:40053: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":410,"thread_start_us":284,"threads_started":1}
I20260928 02:14:23.359694 22546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:14:23.358913 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.235.126:40053 (local address 127.6.235.66:51817)
0928 02:14:23.359426 (+ 513us) negotiation.cc:107] Waiting for socket to connect
0928 02:14:23.359509 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.235.126:40053: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":395,"thread_start_us":276,"threads_started":1}
I20260928 02:14:23.507810 22547 raft_consensus.cc:482] T b6e74e850a0e463f8677dd6a1e1982c4 P 37070660923a4173b9bfe2c5b3abc0c4 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 02:14:23.508040 22547 raft_consensus.cc:504] T b6e74e850a0e463f8677dd6a1e1982c4 P 37070660923a4173b9bfe2c5b3abc0c4 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f11b50d0630c4d91be9b17973fcb0230" member_type: VOTER last_known_addr { host: "127.6.235.67" port: 41399 } } peers { permanent_uuid: "37070660923a4173b9bfe2c5b3abc0c4" member_type: VOTER last_known_addr { host: "127.6.235.65" port: 37261 } } peers { permanent_uuid: "f820529e7c8e4f2f8341f373b1def2d0" member_type: VOTER last_known_addr { host: "127.6.235.66" port: 36697 } }
I20260928 02:14:23.508716 22547 leader_election.cc:288] T b6e74e850a0e463f8677dd6a1e1982c4 P 37070660923a4173b9bfe2c5b3abc0c4 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f11b50d0630c4d91be9b17973fcb0230 (127.6.235.67:41399), f820529e7c8e4f2f8341f373b1def2d0 (127.6.235.66:36697)
W20260928 02:14:23.509886 22145 leader_election.cc:341] T b6e74e850a0e463f8677dd6a1e1982c4 P 37070660923a4173b9bfe2c5b3abc0c4 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer f820529e7c8e4f2f8341f373b1def2d0 (127.6.235.66:36697): Not found: Tablet not found: b6e74e850a0e463f8677dd6a1e1982c4
W20260928 02:14:23.510080 22144 leader_election.cc:341] T b6e74e850a0e463f8677dd6a1e1982c4 P 37070660923a4173b9bfe2c5b3abc0c4 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer f11b50d0630c4d91be9b17973fcb0230 (127.6.235.67:41399): Not found: Tablet not found: b6e74e850a0e463f8677dd6a1e1982c4
I20260928 02:14:23.510200 22144 leader_election.cc:302] T b6e74e850a0e463f8677dd6a1e1982c4 P 37070660923a4173b9bfe2c5b3abc0c4 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 37070660923a4173b9bfe2c5b3abc0c4; no voters: f11b50d0630c4d91be9b17973fcb0230, f820529e7c8e4f2f8341f373b1def2d0
I20260928 02:14:23.510418 22547 raft_consensus.cc:2726] T b6e74e850a0e463f8677dd6a1e1982c4 P 37070660923a4173b9bfe2c5b3abc0c4 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260928 02:14:23.838018 22548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:14:23.837243 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.235.126:40053 (local address 127.6.235.67:39671)
0928 02:14:23.837714 (+ 471us) negotiation.cc:107] Waiting for socket to connect
0928 02:14:23.837799 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.235.126:40053: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":368,"thread_start_us":248,"threads_started":1}
I20260928 02:14:23.895267 22549 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:14:23.894394 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.235.126:40053 (local address 127.6.235.65:41481)
0928 02:14:23.894851 (+ 457us) negotiation.cc:107] Waiting for socket to connect
0928 02:14:23.894953 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.235.126:40053: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":352,"thread_start_us":244,"threads_started":1}
I20260928 02:14:24.361658 22550 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:14:24.360753 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.235.126:40053 (local address 127.6.235.66:50839)
0928 02:14:24.361356 (+ 603us) negotiation.cc:107] Waiting for socket to connect
0928 02:14:24.361456 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.235.126:40053: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":462,"thread_start_us":286,"threads_started":1}
I20260928 02:14:24.839978 22551 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:14:24.839111 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.235.126:40053 (local address 127.6.235.67:48839)
0928 02:14:24.839674 (+ 563us) negotiation.cc:107] Waiting for socket to connect
0928 02:14:24.839772 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.235.126:40053: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":441,"thread_start_us":257,"threads_started":1}
I20260928 02:14:24.897158 22552 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:14:24.896296 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.235.126:40053 (local address 127.6.235.65:35453)
0928 02:14:24.896779 (+ 483us) negotiation.cc:107] Waiting for socket to connect
0928 02:14:24.896879 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.235.126:40053: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":378,"thread_start_us":251,"threads_started":1}
I20260928 02:14:25.363376 22553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:14:25.362600 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.235.126:40053 (local address 127.6.235.66:44837)
0928 02:14:25.363100 (+ 500us) negotiation.cc:107] Waiting for socket to connect
0928 02:14:25.363188 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.235.126:40053: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":399,"thread_start_us":282,"threads_started":1}
I20260928 02:14:25.841842 22554 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:14:25.841104 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.235.126:40053 (local address 127.6.235.67:38631)
0928 02:14:25.841573 (+ 469us) negotiation.cc:107] Waiting for socket to connect
0928 02:14:25.841658 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.235.126:40053: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":367,"thread_start_us":261,"threads_started":1}
I20260928 02:14:25.899117 22555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:14:25.898379 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.235.126:40053 (local address 127.6.235.65:39053)
0928 02:14:25.898844 (+ 465us) negotiation.cc:107] Waiting for socket to connect
0928 02:14:25.898933 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.235.126:40053: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":357,"thread_start_us":238,"threads_started":1}
I20260928 02:14:26.365432 22556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:14:26.364533 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.235.126:40053 (local address 127.6.235.66:59235)
0928 02:14:26.365169 (+ 636us) negotiation.cc:107] Waiting for socket to connect
0928 02:14:26.365273 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.235.126:40053: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":519,"thread_start_us":295,"threads_started":1}
I20260928 02:14:26.454831 22557 raft_consensus.cc:482] T b6e74e850a0e463f8677dd6a1e1982c4 P 37070660923a4173b9bfe2c5b3abc0c4 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 02:14:26.455067 22557 raft_consensus.cc:504] T b6e74e850a0e463f8677dd6a1e1982c4 P 37070660923a4173b9bfe2c5b3abc0c4 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f11b50d0630c4d91be9b17973fcb0230" member_type: VOTER last_known_addr { host: "127.6.235.67" port: 41399 } } peers { permanent_uuid: "37070660923a4173b9bfe2c5b3abc0c4" member_type: VOTER last_known_addr { host: "127.6.235.65" port: 37261 } } peers { permanent_uuid: "f820529e7c8e4f2f8341f373b1def2d0" member_type: VOTER last_known_addr { host: "127.6.235.66" port: 36697 } }
I20260928 02:14:26.455817 22557 leader_election.cc:288] T b6e74e850a0e463f8677dd6a1e1982c4 P 37070660923a4173b9bfe2c5b3abc0c4 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f11b50d0630c4d91be9b17973fcb0230 (127.6.235.67:41399), f820529e7c8e4f2f8341f373b1def2d0 (127.6.235.66:36697)
W20260928 02:14:26.456853 22144 leader_election.cc:341] T b6e74e850a0e463f8677dd6a1e1982c4 P 37070660923a4173b9bfe2c5b3abc0c4 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer f11b50d0630c4d91be9b17973fcb0230 (127.6.235.67:41399): Not found: Tablet not found: b6e74e850a0e463f8677dd6a1e1982c4
W20260928 02:14:26.457099 22145 leader_election.cc:341] T b6e74e850a0e463f8677dd6a1e1982c4 P 37070660923a4173b9bfe2c5b3abc0c4 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer f820529e7c8e4f2f8341f373b1def2d0 (127.6.235.66:36697): Not found: Tablet not found: b6e74e850a0e463f8677dd6a1e1982c4
I20260928 02:14:26.457243 22145 leader_election.cc:302] T b6e74e850a0e463f8677dd6a1e1982c4 P 37070660923a4173b9bfe2c5b3abc0c4 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 37070660923a4173b9bfe2c5b3abc0c4; no voters: f11b50d0630c4d91be9b17973fcb0230, f820529e7c8e4f2f8341f373b1def2d0
I20260928 02:14:26.457572 22557 raft_consensus.cc:2726] T b6e74e850a0e463f8677dd6a1e1982c4 P 37070660923a4173b9bfe2c5b3abc0c4 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260928 02:14:26.844028 22558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:14:26.842926 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.235.126:40053 (local address 127.6.235.67:50895)
0928 02:14:26.843544 (+ 618us) negotiation.cc:107] Waiting for socket to connect
0928 02:14:26.843764 (+ 220us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.235.126:40053: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":448,"thread_start_us":296,"threads_started":1}
I20260928 02:14:26.901382 22559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:14:26.900427 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.235.126:40053 (local address 127.6.235.65:54937)
0928 02:14:26.901020 (+ 593us) negotiation.cc:107] Waiting for socket to connect
0928 02:14:26.901162 (+ 142us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.235.126:40053: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":407,"thread_start_us":260,"threads_started":1}
I20260928 02:14:27.367376 22560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:14:27.366576 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.235.126:40053 (local address 127.6.235.66:51557)
0928 02:14:27.367084 (+ 508us) negotiation.cc:107] Waiting for socket to connect
0928 02:14:27.367187 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.235.126:40053: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":401,"thread_start_us":291,"threads_started":1}
I20260928 02:14:27.846061 22561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:14:27.845156 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.235.126:40053 (local address 127.6.235.67:57839)
0928 02:14:27.845688 (+ 532us) negotiation.cc:107] Waiting for socket to connect
0928 02:14:27.845820 (+ 132us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.235.126:40053: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":375,"thread_start_us":282,"threads_started":1}
I20260928 02:14:27.903316 22562 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:14:27.902433 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.235.126:40053 (local address 127.6.235.65:56923)
0928 02:14:27.903041 (+ 608us) negotiation.cc:107] Waiting for socket to connect
0928 02:14:27.903124 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.235.126:40053: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":464,"thread_start_us":372,"threads_started":1}
I20260928 02:14:28.369262 22563 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:14:28.368406 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.235.126:40053 (local address 127.6.235.66:38353)
0928 02:14:28.368866 (+ 460us) negotiation.cc:107] Waiting for socket to connect
0928 02:14:28.369044 (+ 178us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.235.126:40053: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":353,"thread_start_us":225,"threads_started":1}
I20260928 02:14:28.847985 22564 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:14:28.847117 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.235.126:40053 (local address 127.6.235.67:51625)
0928 02:14:28.847703 (+ 586us) negotiation.cc:107] Waiting for socket to connect
0928 02:14:28.847815 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.235.126:40053: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":443,"thread_start_us":296,"threads_started":1}
I20260928 02:14:28.905056 22565 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:14:28.904224 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.235.126:40053 (local address 127.6.235.65:47961)
0928 02:14:28.904695 (+ 471us) negotiation.cc:107] Waiting for socket to connect
0928 02:14:28.904800 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.235.126:40053: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":371,"thread_start_us":257,"threads_started":1}
I20260928 02:14:29.338376 22566 raft_consensus.cc:482] T b6e74e850a0e463f8677dd6a1e1982c4 P 37070660923a4173b9bfe2c5b3abc0c4 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 02:14:29.338630 22566 raft_consensus.cc:504] T b6e74e850a0e463f8677dd6a1e1982c4 P 37070660923a4173b9bfe2c5b3abc0c4 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f11b50d0630c4d91be9b17973fcb0230" member_type: VOTER last_known_addr { host: "127.6.235.67" port: 41399 } } peers { permanent_uuid: "37070660923a4173b9bfe2c5b3abc0c4" member_type: VOTER last_known_addr { host: "127.6.235.65" port: 37261 } } peers { permanent_uuid: "f820529e7c8e4f2f8341f373b1def2d0" member_type: VOTER last_known_addr { host: "127.6.235.66" port: 36697 } }
I20260928 02:14:29.339313 22566 leader_election.cc:288] T b6e74e850a0e463f8677dd6a1e1982c4 P 37070660923a4173b9bfe2c5b3abc0c4 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f11b50d0630c4d91be9b17973fcb0230 (127.6.235.67:41399), f820529e7c8e4f2f8341f373b1def2d0 (127.6.235.66:36697)
W20260928 02:14:29.340373 22145 leader_election.cc:341] T b6e74e850a0e463f8677dd6a1e1982c4 P 37070660923a4173b9bfe2c5b3abc0c4 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer f820529e7c8e4f2f8341f373b1def2d0 (127.6.235.66:36697): Not found: Tablet not found: b6e74e850a0e463f8677dd6a1e1982c4
W20260928 02:14:29.340598 22144 leader_election.cc:341] T b6e74e850a0e463f8677dd6a1e1982c4 P 37070660923a4173b9bfe2c5b3abc0c4 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer f11b50d0630c4d91be9b17973fcb0230 (127.6.235.67:41399): Not found: Tablet not found: b6e74e850a0e463f8677dd6a1e1982c4
I20260928 02:14:29.340747 22144 leader_election.cc:302] T b6e74e850a0e463f8677dd6a1e1982c4 P 37070660923a4173b9bfe2c5b3abc0c4 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 37070660923a4173b9bfe2c5b3abc0c4; no voters: f11b50d0630c4d91be9b17973fcb0230, f820529e7c8e4f2f8341f373b1def2d0
I20260928 02:14:29.341045 22566 raft_consensus.cc:2726] T b6e74e850a0e463f8677dd6a1e1982c4 P 37070660923a4173b9bfe2c5b3abc0c4 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260928 02:14:29.371273 22567 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:14:29.370518 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.235.126:40053 (local address 127.6.235.66:37745)
0928 02:14:29.371007 (+ 489us) negotiation.cc:107] Waiting for socket to connect
0928 02:14:29.371095 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.235.126:40053: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":385,"thread_start_us":258,"threads_started":1}
I20260928 02:14:29.849933 22568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:14:29.849148 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.235.126:40053 (local address 127.6.235.67:55241)
0928 02:14:29.849625 (+ 477us) negotiation.cc:107] Waiting for socket to connect
0928 02:14:29.849713 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.235.126:40053: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":354,"thread_start_us":236,"threads_started":1}
I20260928 02:14:29.906872 22569 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:14:29.906079 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.235.126:40053 (local address 127.6.235.65:47497)
0928 02:14:29.906546 (+ 467us) negotiation.cc:107] Waiting for socket to connect
0928 02:14:29.906639 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.235.126:40053: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":357,"thread_start_us":221,"threads_started":1}
I20260928 02:14:30.373327 22570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:14:30.372438 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.235.126:40053 (local address 127.6.235.66:50309)
0928 02:14:30.373023 (+ 585us) negotiation.cc:107] Waiting for socket to connect
0928 02:14:30.373145 (+ 122us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.235.126:40053: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":378,"thread_start_us":244,"threads_started":1}
I20260928 02:14:30.851866 22571 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:14:30.851083 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.235.126:40053 (local address 127.6.235.67:47015)
0928 02:14:30.851591 (+ 508us) negotiation.cc:107] Waiting for socket to connect
0928 02:14:30.851698 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.235.126:40053: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":391,"thread_start_us":261,"threads_started":1}
I20260928 02:14:30.908964 22572 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:14:30.908046 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.235.126:40053 (local address 127.6.235.65:59367)
0928 02:14:30.908582 (+ 536us) negotiation.cc:107] Waiting for socket to connect
0928 02:14:30.908672 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.235.126:40053: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":418,"thread_start_us":290,"threads_started":1}
I20260928 02:14:31.375207 22573 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:14:31.374368 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.235.126:40053 (local address 127.6.235.66:46917)
0928 02:14:31.374926 (+ 558us) negotiation.cc:107] Waiting for socket to connect
0928 02:14:31.375011 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.235.126:40053: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":439,"thread_start_us":242,"threads_started":1}
I20260928 02:14:31.853693 22574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:14:31.852866 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.235.126:40053 (local address 127.6.235.67:52797)
0928 02:14:31.853393 (+ 527us) negotiation.cc:107] Waiting for socket to connect
0928 02:14:31.853496 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.235.126:40053: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":426,"thread_start_us":243,"threads_started":1}
I20260928 02:14:31.911043 22575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:14:31.910239 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.235.126:40053 (local address 127.6.235.65:48529)
0928 02:14:31.910773 (+ 534us) negotiation.cc:107] Waiting for socket to connect
0928 02:14:31.910873 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.235.126:40053: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":421,"thread_start_us":249,"threads_started":1}
I20260928 02:14:32.377344 22576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:14:32.376348 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.235.126:40053 (local address 127.6.235.66:47491)
0928 02:14:32.377011 (+ 663us) negotiation.cc:107] Waiting for socket to connect
0928 02:14:32.377158 (+ 147us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.235.126:40053: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":441,"thread_start_us":275,"threads_started":1}
I20260928 02:14:32.830173 7085 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskTlaUuc/build/asan/bin/kudu pid 22094
I20260928 02:14:32.846686 22577 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:14:32.831534 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48456 (local address 127.6.235.65:37261)
0928 02:14:32.832196 (+ 662us) server_negotiation.cc:206] Beginning negotiation
0928 02:14:32.832209 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0928 02:14:32.832247 (+ 38us) server_negotiation.cc:407] Connection header received
0928 02:14:32.832349 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:14:32.832361 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:14:32.832683 (+ 322us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:14:32.832870 (+ 187us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:14:32.834704 (+ 1834us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:14:32.835674 (+ 970us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:14:32.837056 (+ 1382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:14:32.837649 (+ 593us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:14:32.837839 (+ 190us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:14:32.840642 (+ 2803us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:14:32.840655 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:14:32.840670 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:14:32.840706 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:14:32.843503 (+ 2797us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:14:32.844311 (+ 808us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:14:32.844320 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:14:32.844322 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:14:32.844475 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:14:32.844936 (+ 461us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:14:32.844941 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:14:32.844950 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:14:32.845349 (+ 399us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:14:32.845666 (+ 317us) server_negotiation.cc:1035] Waiting for connection context
0928 02:14:32.846022 (+ 356us) server_negotiation.cc:299] Negotiation successful
0928 02:14:32.846334 (+ 312us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":406,"thread_start_us":260,"threads_started":1}
I20260928 02:14:32.848105 22253 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 02:14:32.855821 22579 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:14:32.854975 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.235.126:40053 (local address 127.6.235.67:35107)
0928 02:14:32.855518 (+ 543us) negotiation.cc:107] Waiting for socket to connect
0928 02:14:32.855644 (+ 126us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.235.126:40053: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":425,"thread_start_us":303,"threads_started":1}
I20260928 02:14:32.942725 7085 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTlaUuc/build/asan/bin/kudu with pid 22094
I20260928 02:14:32.963155 7085 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskTlaUuc/build/asan/bin/kudu pid 22261
I20260928 02:14:32.977453 22581 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:14:32.964180 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36496 (local address 127.6.235.66:36697)
0928 02:14:32.964673 (+ 493us) server_negotiation.cc:206] Beginning negotiation
0928 02:14:32.964677 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:14:32.964695 (+ 18us) server_negotiation.cc:407] Connection header received
0928 02:14:32.964752 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:14:32.964756 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:14:32.964817 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:14:32.964981 (+ 164us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:14:32.966390 (+ 1409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:14:32.967214 (+ 824us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:14:32.968495 (+ 1281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:14:32.969056 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:14:32.969199 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:14:32.972027 (+ 2828us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:14:32.972033 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:14:32.972036 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:14:32.972071 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:14:32.974907 (+ 2836us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:14:32.975633 (+ 726us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:14:32.975637 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:14:32.975639 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:14:32.975766 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:14:32.976128 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:14:32.976131 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:14:32.976134 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:14:32.976409 (+ 275us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:14:32.976602 (+ 193us) server_negotiation.cc:1035] Waiting for connection context
0928 02:14:32.976931 (+ 329us) server_negotiation.cc:299] Negotiation successful
0928 02:14:32.977122 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":397,"thread_start_us":283,"threads_started":1}
I20260928 02:14:32.978319 22385 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 02:14:33.063856 7085 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTlaUuc/build/asan/bin/kudu with pid 22261
I20260928 02:14:33.083091 7085 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskTlaUuc/build/asan/bin/kudu pid 22393
I20260928 02:14:33.097522 22583 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:14:33.084008 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47070 (local address 127.6.235.67:41399)
0928 02:14:33.084645 (+ 637us) server_negotiation.cc:206] Beginning negotiation
0928 02:14:33.084649 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:14:33.084666 (+ 17us) server_negotiation.cc:407] Connection header received
0928 02:14:33.084722 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:14:33.084726 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:14:33.084796 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:14:33.084938 (+ 142us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:14:33.086530 (+ 1592us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:14:33.087347 (+ 817us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:14:33.088649 (+ 1302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:14:33.089175 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:14:33.089315 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:14:33.091905 (+ 2590us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:14:33.091910 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:14:33.091913 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:14:33.091949 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:14:33.094895 (+ 2946us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:14:33.095557 (+ 662us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:14:33.095560 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:14:33.095563 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:14:33.095705 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:14:33.096089 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:14:33.096093 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:14:33.096096 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:14:33.096397 (+ 301us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:14:33.096626 (+ 229us) server_negotiation.cc:1035] Waiting for connection context
0928 02:14:33.096968 (+ 342us) server_negotiation.cc:299] Negotiation successful
0928 02:14:33.097220 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":522,"thread_start_us":371,"threads_started":1}
I20260928 02:14:33.098264 22516 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 02:14:33.183836 7085 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTlaUuc/build/asan/bin/kudu with pid 22393
2026-09-28T02:14:33Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedIPKICert (24794 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedTSK
Loading random data
Initializing database '/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790561432251112-7085-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 02:14:33 dist-test-slave-86gv krb5kdc[22587](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 02:14:33 dist-test-slave-86gv krb5kdc[22587](info): set up 2 sockets
Sep 28 02:14:33 dist-test-slave-86gv krb5kdc[22587](info): commencing operation
krb5kdc: starting...
W20260928 02:14:34.314015 7085 mini_kdc.cc:121] Time spent starting KDC: real 1.084s user 0.000s sys 0.028s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790561432251112-7085-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790561432251112-7085-0/krb5kdc/test-user.keytab.
Sep 28 02:14:34 dist-test-slave-86gv krb5kdc[22587](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561674, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T02:14:34Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T02:14:34Z Disabled control of system clock
WARNING: no policy specified for kudu/127.6.235.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.235.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.235.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790561432251112-7085-0/krb5kdc/kudu_127.6.235.126.keytab.
Entry for principal kudu/127.6.235.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790561432251112-7085-0/krb5kdc/kudu_127.6.235.126.keytab.
I20260928 02:14:34.573616 7085 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTlaUuc/build/asan/bin/kudu
/tmp/dist-test-taskTlaUuc/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790561432251112-7085-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790561432251112-7085-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790561432251112-7085-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790561432251112-7085-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.235.126:46725
--webserver_interface=127.6.235.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790561432251112-7085-0/krb5kdc/kudu_127.6.235.126.keytab
--principal=kudu/127.6.235.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.235.84:46687
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.235.126:46725
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790561432251112-7085-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790561432251112-7085-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790561432251112-7085-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:14:34.931185 22600 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:14:34.931620 22600 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:14:34.931808 22600 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:14:34.939159 22600 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260928 02:14:34.939328 22600 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260928 02:14:34.939406 22600 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260928 02:14:34.939486 22600 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 02:14:34.939591 22600 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:14:34.939702 22600 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:14:34.939790 22600 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 02:14:34.939872 22600 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 02:14:34.950162 22600 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.235.84:46687
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790561432251112-7085-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790561432251112-7085-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.6.235.126:46725
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790561432251112-7085-0/krb5kdc/kudu_127.6.235.126.keytab
--principal=kudu/127.6.235.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.235.126:46725
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790561432251112-7085-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.235.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790561432251112-7085-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 00:51:59 UTC on ccycloud.quasar-axgkwn.root.comops.site
build id 4676464
ASAN enabled
I20260928 02:14:34.951646 22600 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:14:34.953243 22600 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:14:34.966454 22608 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:14:34.966521 22606 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:14:34.966543 22605 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:14:34.967180 22600 server_base.cc:1034] running on GCE node
I20260928 02:14:34.968053 22600 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:14:34.969625 22600 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:14:34.970892 22600 hybrid_clock.cc:648] HybridClock initialized: now 1790561674970827 us; error 38 us; skew 500 ppm
I20260928 02:14:34.971287 22600 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 02:14:34 dist-test-slave-86gv krb5kdc[22587](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561674, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.235.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:14:34.974658 22600 init.cc:377] Logged in from keytab as kudu/127.6.235.126@KRBTEST.COM (short username kudu)
I20260928 02:14:34.976181 22600 webserver.cc:460] Webserver started at http://127.6.235.126:46735/ using document root <none> and password file <none>
I20260928 02:14:34.977016 22600 fs_manager.cc:359] Metadata directory not provided
I20260928 02:14:34.977146 22600 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:14:34.977487 22600 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:14:34.980563 22600 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790561432251112-7085-0/minicluster-data/master-0/data/instance:
uuid: "2bc05ba1194e4a02b77ee3f8d3ef5771"
format_stamp: "Formatted at 2026-09-28 02:14:34 on dist-test-slave-86gv"
server_key: "fc8fbdce5fc05aef1c429652b161332b"
server_key_iv: "60e6e9c33bd1190bbbb6c468bc6f8a60"
server_key_version: "encryptionkey@0"
I20260928 02:14:34.981423 22600 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790561432251112-7085-0/minicluster-data/master-0/wal/instance:
uuid: "2bc05ba1194e4a02b77ee3f8d3ef5771"
format_stamp: "Formatted at 2026-09-28 02:14:34 on dist-test-slave-86gv"
server_key: "fc8fbdce5fc05aef1c429652b161332b"
server_key_iv: "60e6e9c33bd1190bbbb6c468bc6f8a60"
server_key_version: "encryptionkey@0"
I20260928 02:14:34.987061 22600 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260928 02:14:34.991058 22615 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:14:34.992563 22600 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260928 02:14:34.992779 22600 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790561432251112-7085-0/minicluster-data/master-0/data,/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790561432251112-7085-0/minicluster-data/master-0/wal
uuid: "2bc05ba1194e4a02b77ee3f8d3ef5771"
format_stamp: "Formatted at 2026-09-28 02:14:34 on dist-test-slave-86gv"
server_key: "fc8fbdce5fc05aef1c429652b161332b"
server_key_iv: "60e6e9c33bd1190bbbb6c468bc6f8a60"
server_key_version: "encryptionkey@0"
I20260928 02:14:34.993101 22600 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790561432251112-7085-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790561432251112-7085-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790561432251112-7085-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:14:35.016961 22600 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:14:35.021931 22600 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:14:35.022224 22600 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:14:35.042099 22600 rpc_server.cc:307] RPC server started. Bound to: 127.6.235.126:46725
I20260928 02:14:35.042125 22666 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.235.126:46725 every 8 connection(s)
I20260928 02:14:35.043598 22600 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790561432251112-7085-0/minicluster-data/master-0/data/info.pb
I20260928 02:14:35.047672 22667 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:14:35.049381 7085 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTlaUuc/build/asan/bin/kudu as pid 22600
I20260928 02:14:35.049600 7085 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790561432251112-7085-0/minicluster-data/master-0/wal/instance
I20260928 02:14:35.050268 7085 external_mini_cluster.cc:1428] Setting key d6a597e475ea70c53668bc789b4b1901
I20260928 02:14:35.063464 22667 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2bc05ba1194e4a02b77ee3f8d3ef5771: Bootstrap starting.
Sep 28 02:14:35 dist-test-slave-86gv krb5kdc[22587](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561674, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.235.126@KRBTEST.COM
I20260928 02:14:35.067041 22667 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2bc05ba1194e4a02b77ee3f8d3ef5771: Neither blocks nor log segments found. Creating new log.
I20260928 02:14:35.068358 22667 log.cc:824] T 00000000000000000000000000000000 P 2bc05ba1194e4a02b77ee3f8d3ef5771: Log is configured to *not* fsync() on all Append() calls
I20260928 02:14:35.072366 22667 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2bc05ba1194e4a02b77ee3f8d3ef5771: No bootstrap required, opened a new log
I20260928 02:14:35.077330 22670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:14:35.052658 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56582 (local address 127.6.235.126:46725)
0928 02:14:35.053715 (+ 1057us) server_negotiation.cc:206] Beginning negotiation
0928 02:14:35.053733 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0928 02:14:35.053791 (+ 58us) server_negotiation.cc:407] Connection header received
0928 02:14:35.055690 (+ 1899us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:14:35.055722 (+ 32us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:14:35.056330 (+ 608us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:14:35.056794 (+ 464us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:14:35.058242 (+ 1448us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:14:35.060366 (+ 2124us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:14:35.062133 (+ 1767us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:14:35.063230 (+ 1097us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:14:35.063425 (+ 195us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:14:35.068075 (+ 4650us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:14:35.068089 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:14:35.068117 (+ 28us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:14:35.068178 (+ 61us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:14:35.072182 (+ 4004us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:14:35.073181 (+ 999us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:14:35.073193 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:14:35.073201 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:14:35.073439 (+ 238us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:14:35.073872 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:14:35.073876 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:14:35.073879 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:14:35.074638 (+ 759us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:14:35.074952 (+ 314us) server_negotiation.cc:1035] Waiting for connection context
0928 02:14:35.075333 (+ 381us) server_negotiation.cc:299] Negotiation successful
0928 02:14:35.075836 (+ 503us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":699,"thread_start_us":489,"threads_started":1}
I20260928 02:14:35.082950 22667 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2bc05ba1194e4a02b77ee3f8d3ef5771 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2bc05ba1194e4a02b77ee3f8d3ef5771" member_type: VOTER last_known_addr { host: "127.6.235.126" port: 46725 } }
I20260928 02:14:35.083333 22667 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2bc05ba1194e4a02b77ee3f8d3ef5771 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:14:35.083406 22667 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2bc05ba1194e4a02b77ee3f8d3ef5771 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2bc05ba1194e4a02b77ee3f8d3ef5771, State: Initialized, Role: FOLLOWER
I20260928 02:14:35.083989 22667 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2bc05ba1194e4a02b77ee3f8d3ef5771 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2bc05ba1194e4a02b77ee3f8d3ef5771" member_type: VOTER last_known_addr { host: "127.6.235.126" port: 46725 } }
I20260928 02:14:35.084206 22667 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2bc05ba1194e4a02b77ee3f8d3ef5771 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 02:14:35.084301 22667 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2bc05ba1194e4a02b77ee3f8d3ef5771 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 02:14:35.084422 22667 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2bc05ba1194e4a02b77ee3f8d3ef5771 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:14:35.086730 22667 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2bc05ba1194e4a02b77ee3f8d3ef5771 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2bc05ba1194e4a02b77ee3f8d3ef5771" member_type: VOTER last_known_addr { host: "127.6.235.126" port: 46725 } }
I20260928 02:14:35.087241 22667 leader_election.cc:302] T 00000000000000000000000000000000 P 2bc05ba1194e4a02b77ee3f8d3ef5771 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2bc05ba1194e4a02b77ee3f8d3ef5771; no voters:
I20260928 02:14:35.087956 22667 leader_election.cc:288] T 00000000000000000000000000000000 P 2bc05ba1194e4a02b77ee3f8d3ef5771 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 02:14:35.088181 22672 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2bc05ba1194e4a02b77ee3f8d3ef5771 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:14:35.089025 22672 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2bc05ba1194e4a02b77ee3f8d3ef5771 [term 1 LEADER]: Becoming Leader. State: Replica: 2bc05ba1194e4a02b77ee3f8d3ef5771, State: Running, Role: LEADER
I20260928 02:14:35.089782 22672 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2bc05ba1194e4a02b77ee3f8d3ef5771 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2bc05ba1194e4a02b77ee3f8d3ef5771" member_type: VOTER last_known_addr { host: "127.6.235.126" port: 46725 } }
I20260928 02:14:35.090481 22667 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2bc05ba1194e4a02b77ee3f8d3ef5771 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 02:14:35.093194 22673 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2bc05ba1194e4a02b77ee3f8d3ef5771 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2bc05ba1194e4a02b77ee3f8d3ef5771" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2bc05ba1194e4a02b77ee3f8d3ef5771" member_type: VOTER last_known_addr { host: "127.6.235.126" port: 46725 } } }
I20260928 02:14:35.093303 22674 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2bc05ba1194e4a02b77ee3f8d3ef5771 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2bc05ba1194e4a02b77ee3f8d3ef5771. Latest consensus state: current_term: 1 leader_uuid: "2bc05ba1194e4a02b77ee3f8d3ef5771" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2bc05ba1194e4a02b77ee3f8d3ef5771" member_type: VOTER last_known_addr { host: "127.6.235.126" port: 46725 } } }
I20260928 02:14:35.093694 22673 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2bc05ba1194e4a02b77ee3f8d3ef5771 [sys.catalog]: This master's current role is: LEADER
I20260928 02:14:35.093698 22674 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2bc05ba1194e4a02b77ee3f8d3ef5771 [sys.catalog]: This master's current role is: LEADER
I20260928 02:14:35.095888 22680 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 02:14:35.102103 22680 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 02:14:35.112231 22680 catalog_manager.cc:1347] Generated new cluster ID: b4531193cd854a578528a141688183d9
I20260928 02:14:35.112375 22680 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 02:14:35.127674 22680 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 02:14:35.129249 22680 catalog_manager.cc:1504] Loading token signing keys...
I20260928 02:14:35.148574 22680 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2bc05ba1194e4a02b77ee3f8d3ef5771: Generated new TSK 0
I20260928 02:14:35.149621 22680 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.6.235.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.235.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.235.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790561432251112-7085-0/krb5kdc/kudu_127.6.235.65.keytab.
Entry for principal kudu/127.6.235.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790561432251112-7085-0/krb5kdc/kudu_127.6.235.65.keytab.
I20260928 02:14:35.228746 7085 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTlaUuc/build/asan/bin/kudu
/tmp/dist-test-taskTlaUuc/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790561432251112-7085-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790561432251112-7085-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790561432251112-7085-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790561432251112-7085-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.235.65:0
--local_ip_for_outbound_sockets=127.6.235.65
--webserver_interface=127.6.235.65
--webserver_port=0
--tserver_master_addrs=127.6.235.126:46725
--keytab_file=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790561432251112-7085-0/krb5kdc/kudu_127.6.235.65.keytab
--principal=kudu/127.6.235.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.235.84:46687
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790561432251112-7085-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790561432251112-7085-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790561432251112-7085-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:14:35.575094 22693 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:14:35.575395 22693 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:14:35.575562 22693 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:14:35.583361 22693 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:14:35.583480 22693 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:14:35.583598 22693 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.235.65
I20260928 02:14:35.594076 22693 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.235.84:46687
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790561432251112-7085-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790561432251112-7085-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790561432251112-7085-0/krb5kdc/kudu_127.6.235.65.keytab
--principal=kudu/127.6.235.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.235.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790561432251112-7085-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.6.235.65
--webserver_port=0
--tserver_master_addrs=127.6.235.126:46725
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.235.65
--log_dir=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790561432251112-7085-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 00:51:59 UTC on ccycloud.quasar-axgkwn.root.comops.site
build id 4676464
ASAN enabled
I20260928 02:14:35.595604 22693 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:14:35.597251 22693 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:14:35.610966 22698 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:14:35.611166 22699 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:14:35.611207 22701 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:14:35.612112 22693 server_base.cc:1034] running on GCE node
I20260928 02:14:35.612843 22693 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:14:35.613960 22693 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:14:35.615170 22693 hybrid_clock.cc:648] HybridClock initialized: now 1790561675615120 us; error 39 us; skew 500 ppm
I20260928 02:14:35.615501 22693 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 02:14:35 dist-test-slave-86gv krb5kdc[22587](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561675, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.235.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:14:35.625566 22693 init.cc:377] Logged in from keytab as kudu/127.6.235.65@KRBTEST.COM (short username kudu)
I20260928 02:14:35.627404 22693 webserver.cc:460] Webserver started at http://127.6.235.65:43859/ using document root <none> and password file <none>
I20260928 02:14:35.628194 22693 fs_manager.cc:359] Metadata directory not provided
I20260928 02:14:35.628352 22693 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:14:35.628692 22693 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:14:35.631845 22693 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790561432251112-7085-0/minicluster-data/ts-0/data/instance:
uuid: "c8d284c745a3487e80f359a662875c9d"
format_stamp: "Formatted at 2026-09-28 02:14:35 on dist-test-slave-86gv"
server_key: "585d0e117c6544ebf3316128d4952e70"
server_key_iv: "edf8eecd2aaa84f5fa0b8e26e372d834"
server_key_version: "encryptionkey@0"
I20260928 02:14:35.632630 22693 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790561432251112-7085-0/minicluster-data/ts-0/wal/instance:
uuid: "c8d284c745a3487e80f359a662875c9d"
format_stamp: "Formatted at 2026-09-28 02:14:35 on dist-test-slave-86gv"
server_key: "585d0e117c6544ebf3316128d4952e70"
server_key_iv: "edf8eecd2aaa84f5fa0b8e26e372d834"
server_key_version: "encryptionkey@0"
I20260928 02:14:35.638094 22693 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.001s
I20260928 02:14:35.641906 22708 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:14:35.643339 22693 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260928 02:14:35.643560 22693 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790561432251112-7085-0/minicluster-data/ts-0/data,/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790561432251112-7085-0/minicluster-data/ts-0/wal
uuid: "c8d284c745a3487e80f359a662875c9d"
format_stamp: "Formatted at 2026-09-28 02:14:35 on dist-test-slave-86gv"
server_key: "585d0e117c6544ebf3316128d4952e70"
server_key_iv: "edf8eecd2aaa84f5fa0b8e26e372d834"
server_key_version: "encryptionkey@0"
I20260928 02:14:35.643811 22693 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790561432251112-7085-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790561432251112-7085-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790561432251112-7085-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:14:35.664136 22693 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:14:35.668839 22693 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:14:35.669200 22693 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:14:35.670218 22693 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:14:35.671670 22693 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:14:35.671816 22693 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:14:35.671973 22693 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:14:35.672053 22693 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:14:35.708607 22693 rpc_server.cc:307] RPC server started. Bound to: 127.6.235.65:33947
I20260928 02:14:35.708644 22820 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.235.65:33947 every 8 connection(s)
I20260928 02:14:35.710014 22693 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790561432251112-7085-0/minicluster-data/ts-0/data/info.pb
I20260928 02:14:35.715421 7085 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTlaUuc/build/asan/bin/kudu as pid 22693
I20260928 02:14:35.715647 7085 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790561432251112-7085-0/minicluster-data/ts-0/wal/instance
I20260928 02:14:35.716495 7085 external_mini_cluster.cc:1428] Setting key 7277243b564f6ec1d91b4b02febf045a
Sep 28 02:14:35 dist-test-slave-86gv krb5kdc[22587](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561675, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.235.65@KRBTEST.COM for kudu/127.6.235.126@KRBTEST.COM
I20260928 02:14:35.731920 22823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:14:35.712638 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.235.65:43093 (local address 127.6.235.126:46725)
0928 02:14:35.713154 (+ 516us) server_negotiation.cc:206] Beginning negotiation
0928 02:14:35.713159 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 02:14:35.714304 (+ 1145us) server_negotiation.cc:407] Connection header received
0928 02:14:35.717045 (+ 2741us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:14:35.717049 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:14:35.717123 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:14:35.717257 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:14:35.718897 (+ 1640us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:14:35.719625 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:14:35.721324 (+ 1699us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:14:35.721731 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:14:35.721889 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:14:35.725870 (+ 3981us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:14:35.725876 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:14:35.725879 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:14:35.725916 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:14:35.728578 (+ 2662us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:14:35.729482 (+ 904us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:14:35.729487 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:14:35.729490 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:14:35.729613 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:14:35.730081 (+ 468us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:14:35.730090 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:14:35.730093 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:14:35.730361 (+ 268us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:14:35.730541 (+ 180us) server_negotiation.cc:1035] Waiting for connection context
0928 02:14:35.731280 (+ 739us) server_negotiation.cc:299] Negotiation successful
0928 02:14:35.731533 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":397,"thread_start_us":293,"threads_started":1}
I20260928 02:14:35.733034 22824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:14:35.713149 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.235.126:46725 (local address 127.6.235.65:43093)
0928 02:14:35.713999 (+ 850us) negotiation.cc:107] Waiting for socket to connect
0928 02:14:35.714045 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0928 02:14:35.716537 (+ 2492us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:14:35.717533 (+ 996us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:14:35.717541 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:14:35.718084 (+ 543us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:14:35.718701 (+ 617us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:14:35.718719 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:14:35.719798 (+ 1079us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:14:35.719804 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:14:35.721169 (+ 1365us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:14:35.721183 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:14:35.721995 (+ 812us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:14:35.722004 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:14:35.722257 (+ 253us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:14:35.723020 (+ 763us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:14:35.723064 (+ 44us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:14:35.725598 (+ 2534us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:14:35.728754 (+ 3156us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:14:35.728764 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:14:35.728785 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:14:35.729330 (+ 545us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:14:35.729735 (+ 405us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:14:35.729740 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:14:35.729743 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:14:35.729951 (+ 208us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:14:35.730499 (+ 548us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:14:35.730508 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:14:35.730893 (+ 385us) client_negotiation.cc:769] Sending connection context
0928 02:14:35.731172 (+ 279us) client_negotiation.cc:240] Negotiation successful
0928 02:14:35.731777 (+ 605us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":619,"thread_start_us":395,"threads_started":1}
I20260928 02:14:35.734777 22821 heartbeater.cc:347] Connected to a master server at 127.6.235.126:46725
I20260928 02:14:35.735098 22821 heartbeater.cc:464] Registering TS with master...
I20260928 02:14:35.735874 22821 heartbeater.cc:511] Master 127.6.235.126:46725 requested a full tablet report, sending...
I20260928 02:14:35.738513 22632 ts_manager.cc:194] Registered new tserver with Master: c8d284c745a3487e80f359a662875c9d (127.6.235.65:33947)
WARNING: no policy specified for kudu/127.6.235.66@KRBTEST.COM; defaulting to no policy
I20260928 02:14:35.740787 22632 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.235.65@KRBTEST.COM'} at 127.6.235.65:43093
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.235.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.235.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790561432251112-7085-0/krb5kdc/kudu_127.6.235.66.keytab.
Entry for principal kudu/127.6.235.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790561432251112-7085-0/krb5kdc/kudu_127.6.235.66.keytab.
I20260928 02:14:35.774685 7085 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTlaUuc/build/asan/bin/kudu
/tmp/dist-test-taskTlaUuc/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790561432251112-7085-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790561432251112-7085-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790561432251112-7085-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790561432251112-7085-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.235.66:0
--local_ip_for_outbound_sockets=127.6.235.66
--webserver_interface=127.6.235.66
--webserver_port=0
--tserver_master_addrs=127.6.235.126:46725
--keytab_file=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790561432251112-7085-0/krb5kdc/kudu_127.6.235.66.keytab
--principal=kudu/127.6.235.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.235.84:46687
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790561432251112-7085-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790561432251112-7085-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790561432251112-7085-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:14:36.127840 22827 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:14:36.128147 22827 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:14:36.128357 22827 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:14:36.135749 22827 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:14:36.135878 22827 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:14:36.136082 22827 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.235.66
I20260928 02:14:36.146397 22827 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.235.84:46687
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790561432251112-7085-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790561432251112-7085-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790561432251112-7085-0/krb5kdc/kudu_127.6.235.66.keytab
--principal=kudu/127.6.235.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.235.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790561432251112-7085-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.6.235.66
--webserver_port=0
--tserver_master_addrs=127.6.235.126:46725
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.235.66
--log_dir=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790561432251112-7085-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 00:51:59 UTC on ccycloud.quasar-axgkwn.root.comops.site
build id 4676464
ASAN enabled
I20260928 02:14:36.147789 22827 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:14:36.149282 22827 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:14:36.163849 22832 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:14:36.163944 22835 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:14:36.163928 22833 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:14:36.164713 22827 server_base.cc:1034] running on GCE node
I20260928 02:14:36.165405 22827 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:14:36.166438 22827 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:14:36.167681 22827 hybrid_clock.cc:648] HybridClock initialized: now 1790561676167633 us; error 43 us; skew 500 ppm
I20260928 02:14:36.168018 22827 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 02:14:36 dist-test-slave-86gv krb5kdc[22587](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561676, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.235.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:14:36.171279 22827 init.cc:377] Logged in from keytab as kudu/127.6.235.66@KRBTEST.COM (short username kudu)
I20260928 02:14:36.172783 22827 webserver.cc:460] Webserver started at http://127.6.235.66:43345/ using document root <none> and password file <none>
I20260928 02:14:36.173543 22827 fs_manager.cc:359] Metadata directory not provided
I20260928 02:14:36.173646 22827 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:14:36.173914 22827 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:14:36.177019 22827 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790561432251112-7085-0/minicluster-data/ts-1/data/instance:
uuid: "6c165d86953345f6a45e9812e72c1b70"
format_stamp: "Formatted at 2026-09-28 02:14:36 on dist-test-slave-86gv"
server_key: "faed87a17ceb68956d872503da8c75b5"
server_key_iv: "ad63fd5a32bacd6faf8da18b56601cbe"
server_key_version: "encryptionkey@0"
I20260928 02:14:36.177774 22827 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790561432251112-7085-0/minicluster-data/ts-1/wal/instance:
uuid: "6c165d86953345f6a45e9812e72c1b70"
format_stamp: "Formatted at 2026-09-28 02:14:36 on dist-test-slave-86gv"
server_key: "faed87a17ceb68956d872503da8c75b5"
server_key_iv: "ad63fd5a32bacd6faf8da18b56601cbe"
server_key_version: "encryptionkey@0"
I20260928 02:14:36.183522 22827 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260928 02:14:36.187793 22842 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:14:36.189621 22827 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.001s
I20260928 02:14:36.189790 22827 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790561432251112-7085-0/minicluster-data/ts-1/data,/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790561432251112-7085-0/minicluster-data/ts-1/wal
uuid: "6c165d86953345f6a45e9812e72c1b70"
format_stamp: "Formatted at 2026-09-28 02:14:36 on dist-test-slave-86gv"
server_key: "faed87a17ceb68956d872503da8c75b5"
server_key_iv: "ad63fd5a32bacd6faf8da18b56601cbe"
server_key_version: "encryptionkey@0"
I20260928 02:14:36.189985 22827 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790561432251112-7085-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790561432251112-7085-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790561432251112-7085-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:14:36.206647 22827 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:14:36.211366 22827 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:14:36.211635 22827 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:14:36.212729 22827 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:14:36.214178 22827 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:14:36.214260 22827 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:14:36.214419 22827 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:14:36.214471 22827 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:14:36.251828 22827 rpc_server.cc:307] RPC server started. Bound to: 127.6.235.66:46099
I20260928 02:14:36.251838 22954 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.235.66:46099 every 8 connection(s)
I20260928 02:14:36.253331 22827 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790561432251112-7085-0/minicluster-data/ts-1/data/info.pb
I20260928 02:14:36.261977 7085 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTlaUuc/build/asan/bin/kudu as pid 22827
I20260928 02:14:36.262184 7085 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790561432251112-7085-0/minicluster-data/ts-1/wal/instance
I20260928 02:14:36.262691 7085 external_mini_cluster.cc:1428] Setting key d0c7ad8b56c142bf47ad0f29f0a65f9f
Sep 28 02:14:36 dist-test-slave-86gv krb5kdc[22587](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561676, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.235.66@KRBTEST.COM for kudu/127.6.235.126@KRBTEST.COM
I20260928 02:14:36.276827 22957 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:14:36.256154 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.235.66:45291 (local address 127.6.235.126:46725)
0928 02:14:36.256721 (+ 567us) server_negotiation.cc:206] Beginning negotiation
0928 02:14:36.256729 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 02:14:36.257732 (+ 1003us) server_negotiation.cc:407] Connection header received
0928 02:14:36.259851 (+ 2119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:14:36.259856 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:14:36.259965 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:14:36.260106 (+ 141us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:14:36.262037 (+ 1931us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:14:36.263021 (+ 984us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:14:36.265157 (+ 2136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:14:36.265655 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:14:36.265822 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:14:36.270393 (+ 4571us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:14:36.270399 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:14:36.270403 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:14:36.270442 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:14:36.273525 (+ 3083us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:14:36.274366 (+ 841us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:14:36.274376 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:14:36.274379 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:14:36.274515 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:14:36.274986 (+ 471us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:14:36.274990 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:14:36.274998 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:14:36.275273 (+ 275us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:14:36.275447 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0928 02:14:36.276194 (+ 747us) server_negotiation.cc:299] Negotiation successful
0928 02:14:36.276438 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":414,"thread_start_us":275,"threads_started":1}
I20260928 02:14:36.277988 22958 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:14:36.256748 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.235.126:46725 (local address 127.6.235.66:45291)
0928 02:14:36.257539 (+ 791us) negotiation.cc:107] Waiting for socket to connect
0928 02:14:36.257590 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0928 02:14:36.259510 (+ 1920us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:14:36.260399 (+ 889us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:14:36.260409 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:14:36.261122 (+ 713us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:14:36.261823 (+ 701us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:14:36.261844 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:14:36.263214 (+ 1370us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:14:36.263219 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:14:36.264993 (+ 1774us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:14:36.265007 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:14:36.265924 (+ 917us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:14:36.265932 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:14:36.266206 (+ 274us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:14:36.267011 (+ 805us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:14:36.267038 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:14:36.270067 (+ 3029us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:14:36.273702 (+ 3635us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:14:36.273714 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:14:36.273740 (+ 26us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:14:36.274205 (+ 465us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:14:36.274638 (+ 433us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:14:36.274643 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:14:36.274648 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:14:36.274856 (+ 208us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:14:36.275408 (+ 552us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:14:36.275417 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:14:36.275822 (+ 405us) client_negotiation.cc:769] Sending connection context
0928 02:14:36.276111 (+ 289us) client_negotiation.cc:240] Negotiation successful
0928 02:14:36.276696 (+ 585us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":526,"thread_start_us":307,"threads_started":1}
I20260928 02:14:36.279719 22955 heartbeater.cc:347] Connected to a master server at 127.6.235.126:46725
I20260928 02:14:36.280088 22955 heartbeater.cc:464] Registering TS with master...
I20260928 02:14:36.281068 22955 heartbeater.cc:511] Master 127.6.235.126:46725 requested a full tablet report, sending...
I20260928 02:14:36.282852 22632 ts_manager.cc:194] Registered new tserver with Master: 6c165d86953345f6a45e9812e72c1b70 (127.6.235.66:46099)
I20260928 02:14:36.283711 22632 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.235.66@KRBTEST.COM'} at 127.6.235.66:45291
WARNING: no policy specified for kudu/127.6.235.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.235.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.235.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790561432251112-7085-0/krb5kdc/kudu_127.6.235.67.keytab.
Entry for principal kudu/127.6.235.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790561432251112-7085-0/krb5kdc/kudu_127.6.235.67.keytab.
I20260928 02:14:36.321034 7085 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTlaUuc/build/asan/bin/kudu
/tmp/dist-test-taskTlaUuc/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790561432251112-7085-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790561432251112-7085-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790561432251112-7085-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790561432251112-7085-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.235.67:0
--local_ip_for_outbound_sockets=127.6.235.67
--webserver_interface=127.6.235.67
--webserver_port=0
--tserver_master_addrs=127.6.235.126:46725
--keytab_file=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790561432251112-7085-0/krb5kdc/kudu_127.6.235.67.keytab
--principal=kudu/127.6.235.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.235.84:46687
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790561432251112-7085-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790561432251112-7085-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790561432251112-7085-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:14:36.671412 22961 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:14:36.671731 22961 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:14:36.671828 22961 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:14:36.679342 22961 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:14:36.679469 22961 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:14:36.679596 22961 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.235.67
I20260928 02:14:36.690245 22961 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.235.84:46687
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790561432251112-7085-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790561432251112-7085-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790561432251112-7085-0/krb5kdc/kudu_127.6.235.67.keytab
--principal=kudu/127.6.235.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.235.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790561432251112-7085-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.6.235.67
--webserver_port=0
--tserver_master_addrs=127.6.235.126:46725
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.235.67
--log_dir=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790561432251112-7085-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 00:51:59 UTC on ccycloud.quasar-axgkwn.root.comops.site
build id 4676464
ASAN enabled
I20260928 02:14:36.691763 22961 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:14:36.693238 22961 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:14:36.707070 22969 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:14:36.707086 22967 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:14:36.707271 22966 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:14:36.708060 22961 server_base.cc:1034] running on GCE node
I20260928 02:14:36.708858 22961 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:14:36.710179 22961 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:14:36.711457 22961 hybrid_clock.cc:648] HybridClock initialized: now 1790561676711376 us; error 69 us; skew 500 ppm
I20260928 02:14:36.711851 22961 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 02:14:36 dist-test-slave-86gv krb5kdc[22587](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561676, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.235.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:14:36.715926 22961 init.cc:377] Logged in from keytab as kudu/127.6.235.67@KRBTEST.COM (short username kudu)
I20260928 02:14:36.717510 22961 webserver.cc:460] Webserver started at http://127.6.235.67:34857/ using document root <none> and password file <none>
I20260928 02:14:36.718194 22961 fs_manager.cc:359] Metadata directory not provided
I20260928 02:14:36.718308 22961 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:14:36.718564 22961 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:14:36.721899 22961 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790561432251112-7085-0/minicluster-data/ts-2/data/instance:
uuid: "613712f91b1348c29b808745f72d275a"
format_stamp: "Formatted at 2026-09-28 02:14:36 on dist-test-slave-86gv"
server_key: "0f80c683ff15b55d4fcfe676df3711df"
server_key_iv: "68eaf86367030440b44df4090297600d"
server_key_version: "encryptionkey@0"
I20260928 02:14:36.722712 22961 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790561432251112-7085-0/minicluster-data/ts-2/wal/instance:
uuid: "613712f91b1348c29b808745f72d275a"
format_stamp: "Formatted at 2026-09-28 02:14:36 on dist-test-slave-86gv"
server_key: "0f80c683ff15b55d4fcfe676df3711df"
server_key_iv: "68eaf86367030440b44df4090297600d"
server_key_version: "encryptionkey@0"
I20260928 02:14:36.728339 22961 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260928 02:14:36.732360 22976 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:14:36.734107 22961 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 02:14:36.734283 22961 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790561432251112-7085-0/minicluster-data/ts-2/data,/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790561432251112-7085-0/minicluster-data/ts-2/wal
uuid: "613712f91b1348c29b808745f72d275a"
format_stamp: "Formatted at 2026-09-28 02:14:36 on dist-test-slave-86gv"
server_key: "0f80c683ff15b55d4fcfe676df3711df"
server_key_iv: "68eaf86367030440b44df4090297600d"
server_key_version: "encryptionkey@0"
I20260928 02:14:36.734500 22961 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790561432251112-7085-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790561432251112-7085-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790561432251112-7085-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:14:36.744474 22821 heartbeater.cc:503] Master 127.6.235.126:46725 was elected leader, sending a full tablet report...
I20260928 02:14:36.759513 22961 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:14:36.764032 22961 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:14:36.764366 22961 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:14:36.765434 22961 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:14:36.766914 22961 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:14:36.766993 22961 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:14:36.767122 22961 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:14:36.767182 22961 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:14:36.804543 22961 rpc_server.cc:307] RPC server started. Bound to: 127.6.235.67:37203
I20260928 02:14:36.804587 23088 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.235.67:37203 every 8 connection(s)
I20260928 02:14:36.805975 22961 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790561432251112-7085-0/minicluster-data/ts-2/data/info.pb
I20260928 02:14:36.807695 7085 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTlaUuc/build/asan/bin/kudu as pid 22961
I20260928 02:14:36.807914 7085 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790561432251112-7085-0/minicluster-data/ts-2/wal/instance
I20260928 02:14:36.808473 7085 external_mini_cluster.cc:1428] Setting key 25aaeca9d53f9f7765e5cc5cf51d3bf5
Sep 28 02:14:36 dist-test-slave-86gv krb5kdc[22587](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561676, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.235.67@KRBTEST.COM for kudu/127.6.235.126@KRBTEST.COM
I20260928 02:14:36.828015 23091 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:14:36.809065 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.235.67:35905 (local address 127.6.235.126:46725)
0928 02:14:36.809578 (+ 513us) server_negotiation.cc:206] Beginning negotiation
0928 02:14:36.809584 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 02:14:36.810886 (+ 1302us) server_negotiation.cc:407] Connection header received
0928 02:14:36.812839 (+ 1953us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:14:36.812843 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:14:36.812988 (+ 145us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:14:36.813095 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:14:36.814660 (+ 1565us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:14:36.815434 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:14:36.817055 (+ 1621us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:14:36.817452 (+ 397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:14:36.817550 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:14:36.821101 (+ 3551us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:14:36.821107 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:14:36.821110 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:14:36.821145 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:14:36.824627 (+ 3482us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:14:36.825498 (+ 871us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:14:36.825504 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:14:36.825508 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:14:36.825695 (+ 187us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:14:36.826151 (+ 456us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:14:36.826156 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:14:36.826160 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:14:36.826455 (+ 295us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:14:36.826636 (+ 181us) server_negotiation.cc:1035] Waiting for connection context
0928 02:14:36.827381 (+ 745us) server_negotiation.cc:299] Negotiation successful
0928 02:14:36.827612 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":385,"thread_start_us":234,"threads_started":1}
I20260928 02:14:36.829190 23092 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:14:36.809774 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.235.126:46725 (local address 127.6.235.67:35905)
0928 02:14:36.810743 (+ 969us) negotiation.cc:107] Waiting for socket to connect
0928 02:14:36.810793 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0928 02:14:36.812569 (+ 1776us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:14:36.813346 (+ 777us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:14:36.813355 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:14:36.813892 (+ 537us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:14:36.814494 (+ 602us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:14:36.814516 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:14:36.815576 (+ 1060us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:14:36.815580 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:14:36.816817 (+ 1237us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:14:36.816830 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:14:36.817675 (+ 845us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:14:36.817680 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:14:36.817885 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:14:36.818523 (+ 638us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:14:36.818545 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:14:36.820805 (+ 2260us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:14:36.824793 (+ 3988us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:14:36.824803 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:14:36.824827 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:14:36.825360 (+ 533us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:14:36.825808 (+ 448us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:14:36.825813 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:14:36.825817 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:14:36.826020 (+ 203us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:14:36.826595 (+ 575us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:14:36.826605 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:14:36.826989 (+ 384us) client_negotiation.cc:769] Sending connection context
0928 02:14:36.827282 (+ 293us) client_negotiation.cc:240] Negotiation successful
0928 02:14:36.827855 (+ 573us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":672,"thread_start_us":423,"threads_started":1}
I20260928 02:14:36.830876 23089 heartbeater.cc:347] Connected to a master server at 127.6.235.126:46725
I20260928 02:14:36.831197 23089 heartbeater.cc:464] Registering TS with master...
I20260928 02:14:36.831952 23089 heartbeater.cc:511] Master 127.6.235.126:46725 requested a full tablet report, sending...
I20260928 02:14:36.833529 22632 ts_manager.cc:194] Registered new tserver with Master: 613712f91b1348c29b808745f72d275a (127.6.235.67:37203)
I20260928 02:14:36.834354 22632 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.235.67@KRBTEST.COM'} at 127.6.235.67:35905
I20260928 02:14:36.837087 7085 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 02:14:36.854456 23091 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:14:36.841598 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56590 (local address 127.6.235.126:46725)
0928 02:14:36.841890 (+ 292us) server_negotiation.cc:206] Beginning negotiation
0928 02:14:36.841897 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 02:14:36.842192 (+ 295us) server_negotiation.cc:407] Connection header received
0928 02:14:36.842394 (+ 202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:14:36.842398 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:14:36.842464 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:14:36.842548 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:14:36.843841 (+ 1293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:14:36.844571 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:14:36.845990 (+ 1419us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:14:36.846390 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:14:36.846487 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:14:36.849166 (+ 2679us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:14:36.849172 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:14:36.849175 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:14:36.849211 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:14:36.851793 (+ 2582us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:14:36.852504 (+ 711us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:14:36.852508 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:14:36.852510 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:14:36.852626 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:14:36.853147 (+ 521us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:14:36.853150 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:14:36.853152 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:14:36.853480 (+ 328us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:14:36.853658 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0928 02:14:36.853937 (+ 279us) server_negotiation.cc:299] Negotiation successful
0928 02:14:36.854119 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260928 02:14:36.858930 22632 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56590:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 02:14:36.866634 22632 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 02:14:36.904106 23104 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:14:36.880290 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41230 (local address 127.6.235.67:37203)
0928 02:14:36.891742 (+ 11452us) server_negotiation.cc:206] Beginning negotiation
0928 02:14:36.891758 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0928 02:14:36.894866 (+ 3108us) server_negotiation.cc:407] Connection header received
0928 02:14:36.894988 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:14:36.894996 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:14:36.895311 (+ 315us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:14:36.896357 (+ 1046us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:14:36.896752 (+ 395us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:14:36.898521 (+ 1769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:14:36.901505 (+ 2984us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:14:36.902895 (+ 1390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:14:36.903058 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:14:36.903302 (+ 244us) server_negotiation.cc:1035] Waiting for connection context
0928 02:14:36.903584 (+ 282us) server_negotiation.cc:299] Negotiation successful
0928 02:14:36.903744 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11140,"thread_start_us":10921,"threads_started":1}
I20260928 02:14:36.904148 23106 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:14:36.879906 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.235.67:37203 (local address 127.0.0.1:41230)
0928 02:14:36.894376 (+ 14470us) negotiation.cc:107] Waiting for socket to connect
0928 02:14:36.894429 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0928 02:14:36.894664 (+ 235us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:14:36.895594 (+ 930us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:14:36.895604 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:14:36.895653 (+ 49us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:14:36.896313 (+ 660us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:14:36.896330 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:14:36.898677 (+ 2347us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:14:36.898683 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:14:36.901306 (+ 2623us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:14:36.901322 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:14:36.903103 (+ 1781us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:14:36.903109 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:14:36.903311 (+ 202us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:14:36.903372 (+ 61us) client_negotiation.cc:769] Sending connection context
0928 02:14:36.903529 (+ 157us) client_negotiation.cc:240] Negotiation successful
0928 02:14:36.903734 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14204,"mutex_wait_us":4,"thread_start_us":16317,"threads_started":1}
I20260928 02:14:36.905498 23102 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:14:36.880609 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.235.65:33947 (local address 127.0.0.1:41070)
0928 02:14:36.896624 (+ 16015us) negotiation.cc:107] Waiting for socket to connect
0928 02:14:36.896644 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0928 02:14:36.896793 (+ 149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:14:36.897814 (+ 1021us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:14:36.897820 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:14:36.897851 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:14:36.898263 (+ 412us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:14:36.898275 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:14:36.899745 (+ 1470us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:14:36.899751 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:14:36.902168 (+ 2417us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:14:36.902187 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:14:36.904443 (+ 2256us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:14:36.904448 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:14:36.904688 (+ 240us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:14:36.904843 (+ 155us) client_negotiation.cc:769] Sending connection context
0928 02:14:36.904952 (+ 109us) client_negotiation.cc:240] Negotiation successful
0928 02:14:36.905083 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15895,"mutex_wait_us":97,"thread_start_us":13372,"threads_started":1}
I20260928 02:14:36.905746 23105 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:14:36.880615 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60796 (local address 127.6.235.66:46099)
0928 02:14:36.893764 (+ 13149us) server_negotiation.cc:206] Beginning negotiation
0928 02:14:36.893775 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 02:14:36.896281 (+ 2506us) server_negotiation.cc:407] Connection header received
0928 02:14:36.896601 (+ 320us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:14:36.896607 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:14:36.896895 (+ 288us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:14:36.897093 (+ 198us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:14:36.898112 (+ 1019us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:14:36.899681 (+ 1569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:14:36.902562 (+ 2881us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:14:36.903917 (+ 1355us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:14:36.904060 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:14:36.904261 (+ 201us) server_negotiation.cc:1035] Waiting for connection context
0928 02:14:36.905259 (+ 998us) server_negotiation.cc:299] Negotiation successful
0928 02:14:36.905448 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12936,"thread_start_us":14246,"threads_started":1}
I20260928 02:14:36.906224 23107 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:14:36.881360 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41070 (local address 127.6.235.65:33947)
0928 02:14:36.895582 (+ 14222us) server_negotiation.cc:206] Beginning negotiation
0928 02:14:36.895590 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 02:14:36.896794 (+ 1204us) server_negotiation.cc:407] Connection header received
0928 02:14:36.897007 (+ 213us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:14:36.897018 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:14:36.897243 (+ 225us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:14:36.897483 (+ 240us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:14:36.898442 (+ 959us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:14:36.899587 (+ 1145us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:14:36.902381 (+ 2794us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:14:36.903916 (+ 1535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:14:36.904058 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:14:36.904282 (+ 224us) server_negotiation.cc:1035] Waiting for connection context
0928 02:14:36.905749 (+ 1467us) server_negotiation.cc:299] Negotiation successful
0928 02:14:36.905930 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14030,"thread_start_us":13851,"threads_started":1}
I20260928 02:14:36.906896 23103 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:14:36.880159 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.235.66:46099 (local address 127.0.0.1:60796)
0928 02:14:36.896044 (+ 15885us) negotiation.cc:107] Waiting for socket to connect
0928 02:14:36.896075 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0928 02:14:36.896401 (+ 326us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:14:36.897391 (+ 990us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:14:36.897399 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:14:36.897432 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:14:36.897871 (+ 439us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:14:36.897886 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:14:36.899863 (+ 1977us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:14:36.899869 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:14:36.902332 (+ 2463us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:14:36.902347 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:14:36.904432 (+ 2085us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:14:36.904438 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:14:36.904689 (+ 251us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:14:36.905080 (+ 391us) client_negotiation.cc:769] Sending connection context
0928 02:14:36.906395 (+ 1315us) client_negotiation.cc:240] Negotiation successful
0928 02:14:36.906538 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15751,"mutex_wait_us":133,"thread_start_us":15625,"threads_started":1}
I20260928 02:14:36.908857 23024 tablet_service.cc:1467] Processing CreateTablet for tablet 53707302e359403fa4ecc7a06b3dba71 (DEFAULT_TABLE table=test-table [id=bb6b493f13ee4e91b0687f384d94585e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:14:36.910548 23024 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 53707302e359403fa4ecc7a06b3dba71. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:14:36.910705 22890 tablet_service.cc:1467] Processing CreateTablet for tablet 53707302e359403fa4ecc7a06b3dba71 (DEFAULT_TABLE table=test-table [id=bb6b493f13ee4e91b0687f384d94585e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:14:36.912388 22890 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 53707302e359403fa4ecc7a06b3dba71. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:14:36.912441 22756 tablet_service.cc:1467] Processing CreateTablet for tablet 53707302e359403fa4ecc7a06b3dba71 (DEFAULT_TABLE table=test-table [id=bb6b493f13ee4e91b0687f384d94585e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:14:36.913709 22756 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 53707302e359403fa4ecc7a06b3dba71. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:14:36.925452 23108 tablet_bootstrap.cc:492] T 53707302e359403fa4ecc7a06b3dba71 P c8d284c745a3487e80f359a662875c9d: Bootstrap starting.
I20260928 02:14:36.929133 23108 tablet_bootstrap.cc:654] T 53707302e359403fa4ecc7a06b3dba71 P c8d284c745a3487e80f359a662875c9d: Neither blocks nor log segments found. Creating new log.
I20260928 02:14:36.931130 23108 log.cc:824] T 53707302e359403fa4ecc7a06b3dba71 P c8d284c745a3487e80f359a662875c9d: Log is configured to *not* fsync() on all Append() calls
I20260928 02:14:36.931236 23110 tablet_bootstrap.cc:492] T 53707302e359403fa4ecc7a06b3dba71 P 6c165d86953345f6a45e9812e72c1b70: Bootstrap starting.
I20260928 02:14:36.933202 23109 tablet_bootstrap.cc:492] T 53707302e359403fa4ecc7a06b3dba71 P 613712f91b1348c29b808745f72d275a: Bootstrap starting.
I20260928 02:14:36.934947 23110 tablet_bootstrap.cc:654] T 53707302e359403fa4ecc7a06b3dba71 P 6c165d86953345f6a45e9812e72c1b70: Neither blocks nor log segments found. Creating new log.
I20260928 02:14:36.936054 23110 log.cc:824] T 53707302e359403fa4ecc7a06b3dba71 P 6c165d86953345f6a45e9812e72c1b70: Log is configured to *not* fsync() on all Append() calls
I20260928 02:14:36.936079 23109 tablet_bootstrap.cc:654] T 53707302e359403fa4ecc7a06b3dba71 P 613712f91b1348c29b808745f72d275a: Neither blocks nor log segments found. Creating new log.
I20260928 02:14:36.937373 23109 log.cc:824] T 53707302e359403fa4ecc7a06b3dba71 P 613712f91b1348c29b808745f72d275a: Log is configured to *not* fsync() on all Append() calls
I20260928 02:14:36.938779 23108 tablet_bootstrap.cc:492] T 53707302e359403fa4ecc7a06b3dba71 P c8d284c745a3487e80f359a662875c9d: No bootstrap required, opened a new log
I20260928 02:14:36.939199 23108 ts_tablet_manager.cc:1397] T 53707302e359403fa4ecc7a06b3dba71 P c8d284c745a3487e80f359a662875c9d: Time spent bootstrapping tablet: real 0.014s user 0.002s sys 0.006s
I20260928 02:14:36.939234 23110 tablet_bootstrap.cc:492] T 53707302e359403fa4ecc7a06b3dba71 P 6c165d86953345f6a45e9812e72c1b70: No bootstrap required, opened a new log
I20260928 02:14:36.939493 23110 ts_tablet_manager.cc:1397] T 53707302e359403fa4ecc7a06b3dba71 P 6c165d86953345f6a45e9812e72c1b70: Time spent bootstrapping tablet: real 0.009s user 0.007s sys 0.000s
I20260928 02:14:36.941308 23109 tablet_bootstrap.cc:492] T 53707302e359403fa4ecc7a06b3dba71 P 613712f91b1348c29b808745f72d275a: No bootstrap required, opened a new log
I20260928 02:14:36.941591 23109 ts_tablet_manager.cc:1397] T 53707302e359403fa4ecc7a06b3dba71 P 613712f91b1348c29b808745f72d275a: Time spent bootstrapping tablet: real 0.009s user 0.007s sys 0.000s
I20260928 02:14:36.946720 23110 raft_consensus.cc:348] T 53707302e359403fa4ecc7a06b3dba71 P 6c165d86953345f6a45e9812e72c1b70 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "613712f91b1348c29b808745f72d275a" member_type: VOTER last_known_addr { host: "127.6.235.67" port: 37203 } } peers { permanent_uuid: "6c165d86953345f6a45e9812e72c1b70" member_type: VOTER last_known_addr { host: "127.6.235.66" port: 46099 } } peers { permanent_uuid: "c8d284c745a3487e80f359a662875c9d" member_type: VOTER last_known_addr { host: "127.6.235.65" port: 33947 } }
I20260928 02:14:36.947130 23110 raft_consensus.cc:374] T 53707302e359403fa4ecc7a06b3dba71 P 6c165d86953345f6a45e9812e72c1b70 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:14:36.947237 23110 raft_consensus.cc:729] T 53707302e359403fa4ecc7a06b3dba71 P 6c165d86953345f6a45e9812e72c1b70 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6c165d86953345f6a45e9812e72c1b70, State: Initialized, Role: FOLLOWER
I20260928 02:14:36.947901 23110 consensus_queue.cc:260] T 53707302e359403fa4ecc7a06b3dba71 P 6c165d86953345f6a45e9812e72c1b70 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "613712f91b1348c29b808745f72d275a" member_type: VOTER last_known_addr { host: "127.6.235.67" port: 37203 } } peers { permanent_uuid: "6c165d86953345f6a45e9812e72c1b70" member_type: VOTER last_known_addr { host: "127.6.235.66" port: 46099 } } peers { permanent_uuid: "c8d284c745a3487e80f359a662875c9d" member_type: VOTER last_known_addr { host: "127.6.235.65" port: 33947 } }
I20260928 02:14:36.949266 22955 heartbeater.cc:503] Master 127.6.235.126:46725 was elected leader, sending a full tablet report...
I20260928 02:14:36.950114 23110 ts_tablet_manager.cc:1428] T 53707302e359403fa4ecc7a06b3dba71 P 6c165d86953345f6a45e9812e72c1b70: Time spent starting tablet: real 0.011s user 0.010s sys 0.000s
I20260928 02:14:36.950237 23108 raft_consensus.cc:348] T 53707302e359403fa4ecc7a06b3dba71 P c8d284c745a3487e80f359a662875c9d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "613712f91b1348c29b808745f72d275a" member_type: VOTER last_known_addr { host: "127.6.235.67" port: 37203 } } peers { permanent_uuid: "6c165d86953345f6a45e9812e72c1b70" member_type: VOTER last_known_addr { host: "127.6.235.66" port: 46099 } } peers { permanent_uuid: "c8d284c745a3487e80f359a662875c9d" member_type: VOTER last_known_addr { host: "127.6.235.65" port: 33947 } }
I20260928 02:14:36.950680 23108 raft_consensus.cc:374] T 53707302e359403fa4ecc7a06b3dba71 P c8d284c745a3487e80f359a662875c9d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:14:36.950768 23108 raft_consensus.cc:729] T 53707302e359403fa4ecc7a06b3dba71 P c8d284c745a3487e80f359a662875c9d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c8d284c745a3487e80f359a662875c9d, State: Initialized, Role: FOLLOWER
I20260928 02:14:36.951442 23108 consensus_queue.cc:260] T 53707302e359403fa4ecc7a06b3dba71 P c8d284c745a3487e80f359a662875c9d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "613712f91b1348c29b808745f72d275a" member_type: VOTER last_known_addr { host: "127.6.235.67" port: 37203 } } peers { permanent_uuid: "6c165d86953345f6a45e9812e72c1b70" member_type: VOTER last_known_addr { host: "127.6.235.66" port: 46099 } } peers { permanent_uuid: "c8d284c745a3487e80f359a662875c9d" member_type: VOTER last_known_addr { host: "127.6.235.65" port: 33947 } }
I20260928 02:14:36.952781 23109 raft_consensus.cc:348] T 53707302e359403fa4ecc7a06b3dba71 P 613712f91b1348c29b808745f72d275a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "613712f91b1348c29b808745f72d275a" member_type: VOTER last_known_addr { host: "127.6.235.67" port: 37203 } } peers { permanent_uuid: "6c165d86953345f6a45e9812e72c1b70" member_type: VOTER last_known_addr { host: "127.6.235.66" port: 46099 } } peers { permanent_uuid: "c8d284c745a3487e80f359a662875c9d" member_type: VOTER last_known_addr { host: "127.6.235.65" port: 33947 } }
I20260928 02:14:36.953265 23109 raft_consensus.cc:374] T 53707302e359403fa4ecc7a06b3dba71 P 613712f91b1348c29b808745f72d275a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:14:36.953362 23109 raft_consensus.cc:729] T 53707302e359403fa4ecc7a06b3dba71 P 613712f91b1348c29b808745f72d275a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 613712f91b1348c29b808745f72d275a, State: Initialized, Role: FOLLOWER
I20260928 02:14:36.954128 23108 ts_tablet_manager.cc:1428] T 53707302e359403fa4ecc7a06b3dba71 P c8d284c745a3487e80f359a662875c9d: Time spent starting tablet: real 0.015s user 0.006s sys 0.009s
I20260928 02:14:36.954111 23109 consensus_queue.cc:260] T 53707302e359403fa4ecc7a06b3dba71 P 613712f91b1348c29b808745f72d275a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "613712f91b1348c29b808745f72d275a" member_type: VOTER last_known_addr { host: "127.6.235.67" port: 37203 } } peers { permanent_uuid: "6c165d86953345f6a45e9812e72c1b70" member_type: VOTER last_known_addr { host: "127.6.235.66" port: 46099 } } peers { permanent_uuid: "c8d284c745a3487e80f359a662875c9d" member_type: VOTER last_known_addr { host: "127.6.235.65" port: 33947 } }
I20260928 02:14:36.955304 23089 heartbeater.cc:503] Master 127.6.235.126:46725 was elected leader, sending a full tablet report...
I20260928 02:14:36.957015 23109 ts_tablet_manager.cc:1428] T 53707302e359403fa4ecc7a06b3dba71 P 613712f91b1348c29b808745f72d275a: Time spent starting tablet: real 0.015s user 0.015s sys 0.000s
W20260928 02:14:36.962517 22822 tablet.cc:2369] T 53707302e359403fa4ecc7a06b3dba71 P c8d284c745a3487e80f359a662875c9d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 02:14:36.980362 23116 raft_consensus.cc:482] T 53707302e359403fa4ecc7a06b3dba71 P 613712f91b1348c29b808745f72d275a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 02:14:36.980753 23116 raft_consensus.cc:504] T 53707302e359403fa4ecc7a06b3dba71 P 613712f91b1348c29b808745f72d275a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "613712f91b1348c29b808745f72d275a" member_type: VOTER last_known_addr { host: "127.6.235.67" port: 37203 } } peers { permanent_uuid: "6c165d86953345f6a45e9812e72c1b70" member_type: VOTER last_known_addr { host: "127.6.235.66" port: 46099 } } peers { permanent_uuid: "c8d284c745a3487e80f359a662875c9d" member_type: VOTER last_known_addr { host: "127.6.235.65" port: 33947 } }
I20260928 02:14:36.982882 23116 leader_election.cc:288] T 53707302e359403fa4ecc7a06b3dba71 P 613712f91b1348c29b808745f72d275a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6c165d86953345f6a45e9812e72c1b70 (127.6.235.66:46099), c8d284c745a3487e80f359a662875c9d (127.6.235.65:33947)
I20260928 02:14:36.989080 23107 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:14:36.983484 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.235.67:45401 (local address 127.6.235.65:33947)
0928 02:14:36.983728 (+ 244us) server_negotiation.cc:206] Beginning negotiation
0928 02:14:36.983732 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:14:36.983756 (+ 24us) server_negotiation.cc:407] Connection header received
0928 02:14:36.983818 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:14:36.983822 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:14:36.983878 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:14:36.983953 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:14:36.984549 (+ 596us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:14:36.985774 (+ 1225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:14:36.987365 (+ 1591us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:14:36.988145 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:14:36.988327 (+ 182us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:14:36.988406 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0928 02:14:36.988665 (+ 259us) server_negotiation.cc:299] Negotiation successful
0928 02:14:36.988793 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":119}
I20260928 02:14:36.989111 23092 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:14:36.983389 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.235.65:33947 (local address 127.6.235.67:45401)
0928 02:14:36.983583 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0928 02:14:36.983605 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0928 02:14:36.983722 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:14:36.984023 (+ 301us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:14:36.984030 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:14:36.984064 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:14:36.984364 (+ 300us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:14:36.984377 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:14:36.985918 (+ 1541us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:14:36.985923 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:14:36.987206 (+ 1283us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:14:36.987218 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:14:36.988312 (+ 1094us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:14:36.988317 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:14:36.988501 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:14:36.988525 (+ 24us) client_negotiation.cc:769] Sending connection context
0928 02:14:36.988639 (+ 114us) client_negotiation.cc:240] Negotiation successful
0928 02:14:36.988770 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":41,"mutex_wait_us":1}
I20260928 02:14:36.989663 23105 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:14:36.983404 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.235.67:58353 (local address 127.6.235.66:46099)
0928 02:14:36.983593 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0928 02:14:36.983601 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 02:14:36.984664 (+ 1063us) server_negotiation.cc:407] Connection header received
0928 02:14:36.984829 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:14:36.984833 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:14:36.984916 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:14:36.984983 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:14:36.985561 (+ 578us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:14:36.986228 (+ 667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:14:36.987909 (+ 1681us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:14:36.988679 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:14:36.988876 (+ 197us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:14:36.988969 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0928 02:14:36.989312 (+ 343us) server_negotiation.cc:299] Negotiation successful
0928 02:14:36.989442 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260928 02:14:36.990268 23117 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:14:36.983408 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.235.66:46099 (local address 127.6.235.67:58353)
0928 02:14:36.984552 (+ 1144us) negotiation.cc:107] Waiting for socket to connect
0928 02:14:36.984569 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0928 02:14:36.984695 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:14:36.985018 (+ 323us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:14:36.985024 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:14:36.985053 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:14:36.985407 (+ 354us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:14:36.985417 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:14:36.986393 (+ 976us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:14:36.986398 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:14:36.987730 (+ 1332us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:14:36.987742 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:14:36.988923 (+ 1181us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:14:36.988930 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:14:36.989083 (+ 153us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:14:36.989109 (+ 26us) client_negotiation.cc:769] Sending connection context
0928 02:14:36.989875 (+ 766us) client_negotiation.cc:240] Negotiation successful
0928 02:14:36.989996 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":986,"mutex_wait_us":382,"thread_start_us":425,"threads_started":1}
I20260928 02:14:36.990177 22776 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "53707302e359403fa4ecc7a06b3dba71" candidate_uuid: "613712f91b1348c29b808745f72d275a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c8d284c745a3487e80f359a662875c9d" is_pre_election: true
I20260928 02:14:36.990852 22776 raft_consensus.cc:2454] T 53707302e359403fa4ecc7a06b3dba71 P c8d284c745a3487e80f359a662875c9d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 613712f91b1348c29b808745f72d275a in term 0.
I20260928 02:14:36.991317 22910 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "53707302e359403fa4ecc7a06b3dba71" candidate_uuid: "613712f91b1348c29b808745f72d275a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6c165d86953345f6a45e9812e72c1b70" is_pre_election: true
I20260928 02:14:36.991593 22978 leader_election.cc:302] T 53707302e359403fa4ecc7a06b3dba71 P 613712f91b1348c29b808745f72d275a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 613712f91b1348c29b808745f72d275a, c8d284c745a3487e80f359a662875c9d; no voters:
I20260928 02:14:36.991894 22910 raft_consensus.cc:2454] T 53707302e359403fa4ecc7a06b3dba71 P 6c165d86953345f6a45e9812e72c1b70 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 613712f91b1348c29b808745f72d275a in term 0.
I20260928 02:14:36.992053 23116 raft_consensus.cc:2781] T 53707302e359403fa4ecc7a06b3dba71 P 613712f91b1348c29b808745f72d275a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 02:14:36.992198 23116 raft_consensus.cc:482] T 53707302e359403fa4ecc7a06b3dba71 P 613712f91b1348c29b808745f72d275a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 02:14:36.992339 23116 raft_consensus.cc:3037] T 53707302e359403fa4ecc7a06b3dba71 P 613712f91b1348c29b808745f72d275a [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:14:36.994722 23116 raft_consensus.cc:504] T 53707302e359403fa4ecc7a06b3dba71 P 613712f91b1348c29b808745f72d275a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "613712f91b1348c29b808745f72d275a" member_type: VOTER last_known_addr { host: "127.6.235.67" port: 37203 } } peers { permanent_uuid: "6c165d86953345f6a45e9812e72c1b70" member_type: VOTER last_known_addr { host: "127.6.235.66" port: 46099 } } peers { permanent_uuid: "c8d284c745a3487e80f359a662875c9d" member_type: VOTER last_known_addr { host: "127.6.235.65" port: 33947 } }
I20260928 02:14:36.995370 23116 leader_election.cc:288] T 53707302e359403fa4ecc7a06b3dba71 P 613712f91b1348c29b808745f72d275a [CANDIDATE]: Term 1 election: Requested vote from peers 6c165d86953345f6a45e9812e72c1b70 (127.6.235.66:46099), c8d284c745a3487e80f359a662875c9d (127.6.235.65:33947)
I20260928 02:14:36.995882 22910 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "53707302e359403fa4ecc7a06b3dba71" candidate_uuid: "613712f91b1348c29b808745f72d275a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6c165d86953345f6a45e9812e72c1b70"
I20260928 02:14:36.996107 22910 raft_consensus.cc:3037] T 53707302e359403fa4ecc7a06b3dba71 P 6c165d86953345f6a45e9812e72c1b70 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:14:36.998598 22910 raft_consensus.cc:2454] T 53707302e359403fa4ecc7a06b3dba71 P 6c165d86953345f6a45e9812e72c1b70 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 613712f91b1348c29b808745f72d275a in term 1.
I20260928 02:14:36.998960 22776 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "53707302e359403fa4ecc7a06b3dba71" candidate_uuid: "613712f91b1348c29b808745f72d275a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c8d284c745a3487e80f359a662875c9d"
I20260928 02:14:36.999042 22980 leader_election.cc:302] T 53707302e359403fa4ecc7a06b3dba71 P 613712f91b1348c29b808745f72d275a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 613712f91b1348c29b808745f72d275a, 6c165d86953345f6a45e9812e72c1b70; no voters:
I20260928 02:14:36.999176 22776 raft_consensus.cc:3037] T 53707302e359403fa4ecc7a06b3dba71 P c8d284c745a3487e80f359a662875c9d [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:14:36.999418 23116 raft_consensus.cc:2781] T 53707302e359403fa4ecc7a06b3dba71 P 613712f91b1348c29b808745f72d275a [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:14:36.999940 23116 raft_consensus.cc:686] T 53707302e359403fa4ecc7a06b3dba71 P 613712f91b1348c29b808745f72d275a [term 1 LEADER]: Becoming Leader. State: Replica: 613712f91b1348c29b808745f72d275a, State: Running, Role: LEADER
I20260928 02:14:37.001103 23116 consensus_queue.cc:237] T 53707302e359403fa4ecc7a06b3dba71 P 613712f91b1348c29b808745f72d275a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "613712f91b1348c29b808745f72d275a" member_type: VOTER last_known_addr { host: "127.6.235.67" port: 37203 } } peers { permanent_uuid: "6c165d86953345f6a45e9812e72c1b70" member_type: VOTER last_known_addr { host: "127.6.235.66" port: 46099 } } peers { permanent_uuid: "c8d284c745a3487e80f359a662875c9d" member_type: VOTER last_known_addr { host: "127.6.235.65" port: 33947 } }
I20260928 02:14:37.002417 22776 raft_consensus.cc:2454] T 53707302e359403fa4ecc7a06b3dba71 P c8d284c745a3487e80f359a662875c9d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 613712f91b1348c29b808745f72d275a in term 1.
W20260928 02:14:37.005681 22956 tablet.cc:2369] T 53707302e359403fa4ecc7a06b3dba71 P 6c165d86953345f6a45e9812e72c1b70: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 02:14:37.006788 22632 catalog_manager.cc:5495] T 53707302e359403fa4ecc7a06b3dba71 P 613712f91b1348c29b808745f72d275a reported cstate change: term changed from 0 to 1, leader changed from <none> to 613712f91b1348c29b808745f72d275a (127.6.235.67). New cstate: current_term: 1 leader_uuid: "613712f91b1348c29b808745f72d275a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "613712f91b1348c29b808745f72d275a" member_type: VOTER last_known_addr { host: "127.6.235.67" port: 37203 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6c165d86953345f6a45e9812e72c1b70" member_type: VOTER last_known_addr { host: "127.6.235.66" port: 46099 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c8d284c745a3487e80f359a662875c9d" member_type: VOTER last_known_addr { host: "127.6.235.65" port: 33947 } health_report { overall_health: UNKNOWN } } }
I20260928 02:14:37.037899 23104 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:14:37.032574 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41234 (local address 127.6.235.67:37203)
0928 02:14:37.032779 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0928 02:14:37.032784 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 02:14:37.032800 (+ 16us) server_negotiation.cc:407] Connection header received
0928 02:14:37.032939 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:14:37.032944 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:14:37.033031 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:14:37.033169 (+ 138us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 02:14:37.033519 (+ 350us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:14:37.034251 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:14:37.035645 (+ 1394us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:14:37.036039 (+ 394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:14:37.036131 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:14:37.036603 (+ 472us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 02:14:37.037359 (+ 756us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 02:14:37.037478 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0928 02:14:37.037579 (+ 101us) server_negotiation.cc:299] Negotiation successful
0928 02:14:37.037651 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260928 02:14:37.046730 22910 raft_consensus.cc:1261] T 53707302e359403fa4ecc7a06b3dba71 P 6c165d86953345f6a45e9812e72c1b70 [term 1 FOLLOWER]: Refusing update from remote peer 613712f91b1348c29b808745f72d275a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 02:14:37.046815 22776 raft_consensus.cc:1261] T 53707302e359403fa4ecc7a06b3dba71 P c8d284c745a3487e80f359a662875c9d [term 1 FOLLOWER]: Refusing update from remote peer 613712f91b1348c29b808745f72d275a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 02:14:37.047873 23116 consensus_queue.cc:1035] T 53707302e359403fa4ecc7a06b3dba71 P 613712f91b1348c29b808745f72d275a [LEADER]: Connected to new peer: Peer: permanent_uuid: "c8d284c745a3487e80f359a662875c9d" member_type: VOTER last_known_addr { host: "127.6.235.65" port: 33947 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 02:14:37.048615 23116 consensus_queue.cc:1035] T 53707302e359403fa4ecc7a06b3dba71 P 613712f91b1348c29b808745f72d275a [LEADER]: Connected to new peer: Peer: permanent_uuid: "6c165d86953345f6a45e9812e72c1b70" member_type: VOTER last_known_addr { host: "127.6.235.66" port: 46099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260928 02:14:37.057601 23090 tablet.cc:2369] T 53707302e359403fa4ecc7a06b3dba71 P 613712f91b1348c29b808745f72d275a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 02:14:37.079111 22631 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56590:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 02:14:37.079450 22631 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56590:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 02:14:37.083200 22631 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 2bc05ba1194e4a02b77ee3f8d3ef5771: Sending DeleteTablet for 3 replicas of tablet 53707302e359403fa4ecc7a06b3dba71
I20260928 02:14:37.084542 23024 tablet_service.cc:1514] Processing DeleteTablet for tablet 53707302e359403fa4ecc7a06b3dba71 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:14:37 UTC) from {username='kudu', principal='kudu/127.6.235.126@KRBTEST.COM'} at 127.0.0.1:41230
I20260928 02:14:37.084538 22756 tablet_service.cc:1514] Processing DeleteTablet for tablet 53707302e359403fa4ecc7a06b3dba71 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:14:37 UTC) from {username='kudu', principal='kudu/127.6.235.126@KRBTEST.COM'} at 127.0.0.1:41070
I20260928 02:14:37.084540 22890 tablet_service.cc:1514] Processing DeleteTablet for tablet 53707302e359403fa4ecc7a06b3dba71 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:14:37 UTC) from {username='kudu', principal='kudu/127.6.235.126@KRBTEST.COM'} at 127.0.0.1:60796
I20260928 02:14:37.090253 7085 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskTlaUuc/build/asan/bin/kudu pid 22693
I20260928 02:14:37.102602 23130 tablet_replica.cc:330] T 53707302e359403fa4ecc7a06b3dba71 P 6c165d86953345f6a45e9812e72c1b70: stopping tablet replica
I20260928 02:14:37.103065 23130 raft_consensus.cc:2229] T 53707302e359403fa4ecc7a06b3dba71 P 6c165d86953345f6a45e9812e72c1b70 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 02:14:37.103576 23130 raft_consensus.cc:2258] T 53707302e359403fa4ecc7a06b3dba71 P 6c165d86953345f6a45e9812e72c1b70 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 02:14:37.104386 23131 tablet_replica.cc:330] T 53707302e359403fa4ecc7a06b3dba71 P c8d284c745a3487e80f359a662875c9d: stopping tablet replica
I20260928 02:14:37.104791 23131 raft_consensus.cc:2229] T 53707302e359403fa4ecc7a06b3dba71 P c8d284c745a3487e80f359a662875c9d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 02:14:37.105257 23131 raft_consensus.cc:2258] T 53707302e359403fa4ecc7a06b3dba71 P c8d284c745a3487e80f359a662875c9d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 02:14:37.107506 23131 ts_tablet_manager.cc:1905] T 53707302e359403fa4ecc7a06b3dba71 P c8d284c745a3487e80f359a662875c9d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 02:14:37.107827 23132 tablet_replica.cc:330] T 53707302e359403fa4ecc7a06b3dba71 P 613712f91b1348c29b808745f72d275a: stopping tablet replica
I20260928 02:14:37.108281 23132 raft_consensus.cc:2229] T 53707302e359403fa4ecc7a06b3dba71 P 613712f91b1348c29b808745f72d275a [term 1 LEADER]: Raft consensus shutting down.
I20260928 02:14:37.108994 23132 raft_consensus.cc:2258] T 53707302e359403fa4ecc7a06b3dba71 P 613712f91b1348c29b808745f72d275a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 02:14:37.111460 23132 ts_tablet_manager.cc:1905] T 53707302e359403fa4ecc7a06b3dba71 P 613712f91b1348c29b808745f72d275a: Deleting tablet data with delete state TABLET_DATA_DELETED
Sep 28 02:14:37 dist-test-slave-86gv krb5kdc[22587](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561674, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.235.65@KRBTEST.COM
I20260928 02:14:37.115489 23131 ts_tablet_manager.cc:1918] T 53707302e359403fa4ecc7a06b3dba71 P c8d284c745a3487e80f359a662875c9d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 02:14:37.115688 23131 log.cc:1196] T 53707302e359403fa4ecc7a06b3dba71 P c8d284c745a3487e80f359a662875c9d: Deleting WAL directory at /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790561432251112-7085-0/minicluster-data/ts-0/wal/wals/53707302e359403fa4ecc7a06b3dba71
I20260928 02:14:37.116219 23131 ts_tablet_manager.cc:1939] T 53707302e359403fa4ecc7a06b3dba71 P c8d284c745a3487e80f359a662875c9d: Deleting consensus metadata
I20260928 02:14:37.118881 23130 ts_tablet_manager.cc:1905] T 53707302e359403fa4ecc7a06b3dba71 P 6c165d86953345f6a45e9812e72c1b70: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 02:14:37.119153 23132 ts_tablet_manager.cc:1918] T 53707302e359403fa4ecc7a06b3dba71 P 613712f91b1348c29b808745f72d275a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 02:14:37.119339 23132 log.cc:1196] T 53707302e359403fa4ecc7a06b3dba71 P 613712f91b1348c29b808745f72d275a: Deleting WAL directory at /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790561432251112-7085-0/minicluster-data/ts-2/wal/wals/53707302e359403fa4ecc7a06b3dba71
I20260928 02:14:37.119899 23132 ts_tablet_manager.cc:1939] T 53707302e359403fa4ecc7a06b3dba71 P 613712f91b1348c29b808745f72d275a: Deleting consensus metadata
I20260928 02:14:37.121850 22616 catalog_manager.cc:4841] TS 613712f91b1348c29b808745f72d275a (127.6.235.67:37203): tablet 53707302e359403fa4ecc7a06b3dba71 (table test-table [id=bb6b493f13ee4e91b0687f384d94585e]) successfully deleted
I20260928 02:14:37.122471 23107 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:14:37.091440 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41080 (local address 127.6.235.65:33947)
0928 02:14:37.091644 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0928 02:14:37.091652 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 02:14:37.095034 (+ 3382us) server_negotiation.cc:407] Connection header received
0928 02:14:37.095765 (+ 731us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:14:37.095772 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:14:37.095873 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:14:37.095976 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:14:37.100668 (+ 4692us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:14:37.101986 (+ 1318us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:14:37.106103 (+ 4117us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:14:37.106775 (+ 672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:14:37.106930 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:14:37.113126 (+ 6196us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:14:37.113142 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:14:37.113163 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:14:37.113216 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:14:37.118144 (+ 4928us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:14:37.119526 (+ 1382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:14:37.119536 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:14:37.119539 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:14:37.119719 (+ 180us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:14:37.120280 (+ 561us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:14:37.120285 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:14:37.120288 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:14:37.120725 (+ 437us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:14:37.121044 (+ 319us) server_negotiation.cc:1035] Waiting for connection context
0928 02:14:37.121761 (+ 717us) server_negotiation.cc:299] Negotiation successful
0928 02:14:37.122065 (+ 304us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260928 02:14:37.123519 22617 catalog_manager.cc:4841] TS c8d284c745a3487e80f359a662875c9d (127.6.235.65:33947): tablet 53707302e359403fa4ecc7a06b3dba71 (table test-table [id=bb6b493f13ee4e91b0687f384d94585e]) successfully deleted
I20260928 02:14:37.124817 22816 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 02:14:37.126605 23130 ts_tablet_manager.cc:1918] T 53707302e359403fa4ecc7a06b3dba71 P 6c165d86953345f6a45e9812e72c1b70: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 02:14:37.126811 23130 log.cc:1196] T 53707302e359403fa4ecc7a06b3dba71 P 6c165d86953345f6a45e9812e72c1b70: Deleting WAL directory at /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790561432251112-7085-0/minicluster-data/ts-1/wal/wals/53707302e359403fa4ecc7a06b3dba71
I20260928 02:14:37.127504 23130 ts_tablet_manager.cc:1939] T 53707302e359403fa4ecc7a06b3dba71 P 6c165d86953345f6a45e9812e72c1b70: Deleting consensus metadata
I20260928 02:14:37.129817 22619 catalog_manager.cc:4841] TS 6c165d86953345f6a45e9812e72c1b70 (127.6.235.66:46099): tablet 53707302e359403fa4ecc7a06b3dba71 (table test-table [id=bb6b493f13ee4e91b0687f384d94585e]) successfully deleted
I20260928 02:14:37.234000 7085 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTlaUuc/build/asan/bin/kudu with pid 22693
I20260928 02:14:37.255596 7085 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskTlaUuc/build/asan/bin/kudu pid 22827
Sep 28 02:14:37 dist-test-slave-86gv krb5kdc[22587](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561674, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.235.66@KRBTEST.COM
I20260928 02:14:37.272804 23105 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:14:37.256550 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60806 (local address 127.6.235.66:46099)
0928 02:14:37.256735 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0928 02:14:37.256742 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 02:14:37.256862 (+ 120us) server_negotiation.cc:407] Connection header received
0928 02:14:37.257096 (+ 234us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:14:37.257102 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:14:37.257174 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:14:37.257291 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:14:37.258595 (+ 1304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:14:37.259630 (+ 1035us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:14:37.261232 (+ 1602us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:14:37.261652 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:14:37.261795 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:14:37.266423 (+ 4628us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:14:37.266436 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:14:37.266452 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:14:37.266489 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:14:37.269785 (+ 3296us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:14:37.270564 (+ 779us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:14:37.270573 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:14:37.270575 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:14:37.270707 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:14:37.271132 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:14:37.271141 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:14:37.271144 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:14:37.271537 (+ 393us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:14:37.271863 (+ 326us) server_negotiation.cc:1035] Waiting for connection context
0928 02:14:37.272199 (+ 336us) server_negotiation.cc:299] Negotiation successful
0928 02:14:37.272424 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260928 02:14:37.273793 22950 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 02:14:37.366575 7085 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTlaUuc/build/asan/bin/kudu with pid 22827
I20260928 02:14:37.386394 7085 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskTlaUuc/build/asan/bin/kudu pid 22961
Sep 28 02:14:37 dist-test-slave-86gv krb5kdc[22587](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561674, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.235.67@KRBTEST.COM
I20260928 02:14:37.404871 23104 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:14:37.387273 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41250 (local address 127.6.235.67:37203)
0928 02:14:37.387455 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0928 02:14:37.387461 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 02:14:37.387513 (+ 52us) server_negotiation.cc:407] Connection header received
0928 02:14:37.387859 (+ 346us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:14:37.387865 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:14:37.387952 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:14:37.388071 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:14:37.389539 (+ 1468us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:14:37.390384 (+ 845us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:14:37.391815 (+ 1431us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:14:37.392340 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:14:37.392490 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:14:37.397460 (+ 4970us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:14:37.397477 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:14:37.397494 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:14:37.397543 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:14:37.401542 (+ 3999us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:14:37.402375 (+ 833us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:14:37.402390 (+ 15us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:14:37.402393 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:14:37.402592 (+ 199us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:14:37.403020 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:14:37.403025 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:14:37.403029 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:14:37.403438 (+ 409us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:14:37.403783 (+ 345us) server_negotiation.cc:1035] Waiting for connection context
0928 02:14:37.404147 (+ 364us) server_negotiation.cc:299] Negotiation successful
0928 02:14:37.404398 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260928 02:14:37.405663 23084 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 02:14:37.532311 7085 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTlaUuc/build/asan/bin/kudu with pid 22961
I20260928 02:14:37.553351 7085 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskTlaUuc/build/asan/bin/kudu pid 22600
I20260928 02:14:37.554586 22662 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 02:14:37.668783 7085 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTlaUuc/build/asan/bin/kudu with pid 22600
I20260928 02:14:57.691781 7085 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTlaUuc/build/asan/bin/kudu
/tmp/dist-test-taskTlaUuc/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790561432251112-7085-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790561432251112-7085-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790561432251112-7085-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790561432251112-7085-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.235.126:46725
--webserver_interface=127.6.235.126
--webserver_port=46735
--keytab_file=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790561432251112-7085-0/krb5kdc/kudu_127.6.235.126.keytab
--principal=kudu/127.6.235.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.235.84:46687
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.235.126:46725
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--tsk_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790561432251112-7085-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790561432251112-7085-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790561432251112-7085-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:14:58.053277 23138 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:14:58.053772 23138 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:14:58.053947 23138 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:14:58.061496 23138 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260928 02:14:58.061654 23138 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260928 02:14:58.061728 23138 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260928 02:14:58.061808 23138 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 02:14:58.061890 23138 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:14:58.061993 23138 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:14:58.062086 23138 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 02:14:58.062166 23138 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 02:14:58.072340 23138 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.235.84:46687
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790561432251112-7085-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790561432251112-7085-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_private_key_password_cmd=echo foo
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.6.235.126:46725
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790561432251112-7085-0/krb5kdc/kudu_127.6.235.126.keytab
--principal=kudu/127.6.235.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.235.126:46725
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790561432251112-7085-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.235.126
--webserver_port=46735
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790561432251112-7085-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 00:51:59 UTC on ccycloud.quasar-axgkwn.root.comops.site
build id 4676464
ASAN enabled
I20260928 02:14:58.073812 23138 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:14:58.075197 23138 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:14:58.099474 23144 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:14:58.099439 23145 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:14:58.099501 23147 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:14:58.102627 23138 server_base.cc:1034] running on GCE node
I20260928 02:14:58.103662 23138 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:14:58.105475 23138 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:14:58.106777 23138 hybrid_clock.cc:648] HybridClock initialized: now 1790561698106700 us; error 47 us; skew 500 ppm
I20260928 02:14:58.107230 23138 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 02:14:58 dist-test-slave-86gv krb5kdc[22587](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561698, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.235.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:14:58.117615 23138 init.cc:377] Logged in from keytab as kudu/127.6.235.126@KRBTEST.COM (short username kudu)
I20260928 02:14:58.119678 23138 webserver.cc:460] Webserver started at http://127.6.235.126:46735/ using document root <none> and password file <none>
I20260928 02:14:58.120569 23138 fs_manager.cc:359] Metadata directory not provided
I20260928 02:14:58.120718 23138 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:14:58.127074 23138 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20260928 02:14:58.130388 23154 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:14:58.132246 23138 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260928 02:14:58.132511 23138 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790561432251112-7085-0/minicluster-data/master-0/data,/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790561432251112-7085-0/minicluster-data/master-0/wal
uuid: "2bc05ba1194e4a02b77ee3f8d3ef5771"
format_stamp: "Formatted at 2026-09-28 02:14:34 on dist-test-slave-86gv"
server_key: "fc8fbdce5fc05aef1c429652b161332b"
server_key_iv: "60e6e9c33bd1190bbbb6c468bc6f8a60"
server_key_version: "encryptionkey@0"
I20260928 02:14:58.133845 23138 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790561432251112-7085-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790561432251112-7085-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790561432251112-7085-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:14:58.150323 23138 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:14:58.154995 23138 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:14:58.155315 23138 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:14:58.176919 23138 rpc_server.cc:307] RPC server started. Bound to: 127.6.235.126:46725
I20260928 02:14:58.177013 23205 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.235.126:46725 every 8 connection(s)
I20260928 02:14:58.178660 23138 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790561432251112-7085-0/minicluster-data/master-0/data/info.pb
I20260928 02:14:58.179687 7085 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTlaUuc/build/asan/bin/kudu as pid 23138
I20260928 02:14:58.180604 7085 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTlaUuc/build/asan/bin/kudu
/tmp/dist-test-taskTlaUuc/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790561432251112-7085-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790561432251112-7085-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790561432251112-7085-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790561432251112-7085-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.235.65:33947
--local_ip_for_outbound_sockets=127.6.235.65
--tserver_master_addrs=127.6.235.126:46725
--webserver_port=43859
--webserver_interface=127.6.235.65
--keytab_file=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790561432251112-7085-0/krb5kdc/kudu_127.6.235.65.keytab
--principal=kudu/127.6.235.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.235.84:46687
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790561432251112-7085-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790561432251112-7085-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790561432251112-7085-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 02:14:58.186843 23206 sys_catalog.cc:263] Verifying existing consensus state
I20260928 02:14:58.191358 23206 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2bc05ba1194e4a02b77ee3f8d3ef5771: Bootstrap starting.
I20260928 02:14:58.217685 23206 log.cc:824] T 00000000000000000000000000000000 P 2bc05ba1194e4a02b77ee3f8d3ef5771: Log is configured to *not* fsync() on all Append() calls
I20260928 02:14:58.230225 23206 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2bc05ba1194e4a02b77ee3f8d3ef5771: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 02:14:58.230813 23206 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2bc05ba1194e4a02b77ee3f8d3ef5771: Bootstrap complete.
I20260928 02:14:58.241606 23206 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2bc05ba1194e4a02b77ee3f8d3ef5771 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2bc05ba1194e4a02b77ee3f8d3ef5771" member_type: VOTER last_known_addr { host: "127.6.235.126" port: 46725 } }
I20260928 02:14:58.242666 23206 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2bc05ba1194e4a02b77ee3f8d3ef5771 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2bc05ba1194e4a02b77ee3f8d3ef5771, State: Initialized, Role: FOLLOWER
I20260928 02:14:58.243422 23206 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2bc05ba1194e4a02b77ee3f8d3ef5771 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2bc05ba1194e4a02b77ee3f8d3ef5771" member_type: VOTER last_known_addr { host: "127.6.235.126" port: 46725 } }
I20260928 02:14:58.243716 23206 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2bc05ba1194e4a02b77ee3f8d3ef5771 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 02:14:58.243854 23206 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2bc05ba1194e4a02b77ee3f8d3ef5771 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 02:14:58.244043 23206 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2bc05ba1194e4a02b77ee3f8d3ef5771 [term 1 FOLLOWER]: Advancing to term 2
I20260928 02:14:58.248297 23206 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2bc05ba1194e4a02b77ee3f8d3ef5771 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2bc05ba1194e4a02b77ee3f8d3ef5771" member_type: VOTER last_known_addr { host: "127.6.235.126" port: 46725 } }
I20260928 02:14:58.248927 23206 leader_election.cc:302] T 00000000000000000000000000000000 P 2bc05ba1194e4a02b77ee3f8d3ef5771 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2bc05ba1194e4a02b77ee3f8d3ef5771; no voters:
I20260928 02:14:58.249609 23206 leader_election.cc:288] T 00000000000000000000000000000000 P 2bc05ba1194e4a02b77ee3f8d3ef5771 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260928 02:14:58.249789 23210 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2bc05ba1194e4a02b77ee3f8d3ef5771 [term 2 FOLLOWER]: Leader election won for term 2
I20260928 02:14:58.251031 23210 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2bc05ba1194e4a02b77ee3f8d3ef5771 [term 2 LEADER]: Becoming Leader. State: Replica: 2bc05ba1194e4a02b77ee3f8d3ef5771, State: Running, Role: LEADER
I20260928 02:14:58.251734 23210 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2bc05ba1194e4a02b77ee3f8d3ef5771 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2bc05ba1194e4a02b77ee3f8d3ef5771" member_type: VOTER last_known_addr { host: "127.6.235.126" port: 46725 } }
I20260928 02:14:58.252163 23206 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2bc05ba1194e4a02b77ee3f8d3ef5771 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 02:14:58.254530 23212 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2bc05ba1194e4a02b77ee3f8d3ef5771 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2bc05ba1194e4a02b77ee3f8d3ef5771. Latest consensus state: current_term: 2 leader_uuid: "2bc05ba1194e4a02b77ee3f8d3ef5771" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2bc05ba1194e4a02b77ee3f8d3ef5771" member_type: VOTER last_known_addr { host: "127.6.235.126" port: 46725 } } }
I20260928 02:14:58.254959 23212 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2bc05ba1194e4a02b77ee3f8d3ef5771 [sys.catalog]: This master's current role is: LEADER
I20260928 02:14:58.255611 23211 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2bc05ba1194e4a02b77ee3f8d3ef5771 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "2bc05ba1194e4a02b77ee3f8d3ef5771" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2bc05ba1194e4a02b77ee3f8d3ef5771" member_type: VOTER last_known_addr { host: "127.6.235.126" port: 46725 } } }
I20260928 02:14:58.255949 23211 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2bc05ba1194e4a02b77ee3f8d3ef5771 [sys.catalog]: This master's current role is: LEADER
I20260928 02:14:58.265166 23218 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 02:14:58.275195 23218 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 02:14:58.276934 23218 catalog_manager.cc:1259] Loaded cluster ID: b4531193cd854a578528a141688183d9
I20260928 02:14:58.277122 23218 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 02:14:58.282320 23218 catalog_manager.cc:1504] Loading token signing keys...
I20260928 02:14:58.284574 23218 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 2bc05ba1194e4a02b77ee3f8d3ef5771: Loaded TSK: 0
I20260928 02:14:58.317415 23227 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20260928 02:14:58.320274 23218 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2bc05ba1194e4a02b77ee3f8d3ef5771: Generated new TSK 1
I20260928 02:14:58.320712 23218 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260928 02:14:58.570931 23208 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:14:58.571293 23208 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:14:58.571460 23208 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:14:58.579279 23208 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:14:58.579496 23208 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:14:58.579692 23208 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.235.65
I20260928 02:14:58.590106 23208 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.235.84:46687
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790561432251112-7085-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790561432251112-7085-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790561432251112-7085-0/krb5kdc/kudu_127.6.235.65.keytab
--principal=kudu/127.6.235.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.235.65:33947
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790561432251112-7085-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.6.235.65
--webserver_port=43859
--tserver_master_addrs=127.6.235.126:46725
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.235.65
--log_dir=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790561432251112-7085-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 00:51:59 UTC on ccycloud.quasar-axgkwn.root.comops.site
build id 4676464
ASAN enabled
I20260928 02:14:58.591545 23208 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:14:58.592969 23208 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:14:58.606961 23234 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:14:58.607594 23236 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:14:58.607702 23233 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:14:58.608088 23208 server_base.cc:1034] running on GCE node
I20260928 02:14:58.609062 23208 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:14:58.610489 23208 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:14:58.611789 23208 hybrid_clock.cc:648] HybridClock initialized: now 1790561698611709 us; error 69 us; skew 500 ppm
I20260928 02:14:58.612215 23208 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 02:14:58 dist-test-slave-86gv krb5kdc[22587](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561698, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.235.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:14:58.617007 23208 init.cc:377] Logged in from keytab as kudu/127.6.235.65@KRBTEST.COM (short username kudu)
I20260928 02:14:58.619185 23208 webserver.cc:460] Webserver started at http://127.6.235.65:43859/ using document root <none> and password file <none>
I20260928 02:14:58.620252 23208 fs_manager.cc:359] Metadata directory not provided
I20260928 02:14:58.620434 23208 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:14:58.628357 23208 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.006s sys 0.000s
I20260928 02:14:58.632439 23243 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:14:58.634497 23208 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.000s sys 0.003s
I20260928 02:14:58.634696 23208 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790561432251112-7085-0/minicluster-data/ts-0/data,/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790561432251112-7085-0/minicluster-data/ts-0/wal
uuid: "c8d284c745a3487e80f359a662875c9d"
format_stamp: "Formatted at 2026-09-28 02:14:35 on dist-test-slave-86gv"
server_key: "585d0e117c6544ebf3316128d4952e70"
server_key_iv: "edf8eecd2aaa84f5fa0b8e26e372d834"
server_key_version: "encryptionkey@0"
I20260928 02:14:58.636315 23208 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790561432251112-7085-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790561432251112-7085-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790561432251112-7085-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:14:58.689373 23208 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:14:58.695338 23208 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:14:58.695604 23208 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:14:58.696718 23208 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:14:58.698333 23208 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:14:58.698438 23208 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:14:58.698529 23208 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:14:58.698581 23208 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:14:58.736872 23208 rpc_server.cc:307] RPC server started. Bound to: 127.6.235.65:33947
I20260928 02:14:58.736981 23355 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.235.65:33947 every 8 connection(s)
I20260928 02:14:58.738729 23208 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790561432251112-7085-0/minicluster-data/ts-0/data/info.pb
I20260928 02:14:58.743110 7085 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTlaUuc/build/asan/bin/kudu as pid 23208
I20260928 02:14:58.744004 7085 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTlaUuc/build/asan/bin/kudu
/tmp/dist-test-taskTlaUuc/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790561432251112-7085-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790561432251112-7085-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790561432251112-7085-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790561432251112-7085-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.235.66:46099
--local_ip_for_outbound_sockets=127.6.235.66
--tserver_master_addrs=127.6.235.126:46725
--webserver_port=43345
--webserver_interface=127.6.235.66
--keytab_file=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790561432251112-7085-0/krb5kdc/kudu_127.6.235.66.keytab
--principal=kudu/127.6.235.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.235.84:46687
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790561432251112-7085-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790561432251112-7085-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790561432251112-7085-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 28 02:14:58 dist-test-slave-86gv krb5kdc[22587](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561698, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.235.65@KRBTEST.COM for kudu/127.6.235.126@KRBTEST.COM
I20260928 02:14:58.767359 23359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:14:58.743052 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.235.126:46725 (local address 127.6.235.65:60259)
0928 02:14:58.744000 (+ 948us) negotiation.cc:107] Waiting for socket to connect
0928 02:14:58.744061 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0928 02:14:58.746182 (+ 2121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:14:58.747821 (+ 1639us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:14:58.747832 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:14:58.748483 (+ 651us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:14:58.749167 (+ 684us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:14:58.749188 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:14:58.750883 (+ 1695us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:14:58.750887 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:14:58.752242 (+ 1355us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:14:58.752255 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:14:58.754013 (+ 1758us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:14:58.754019 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:14:58.754243 (+ 224us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:14:58.755032 (+ 789us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:14:58.755068 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:14:58.757534 (+ 2466us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:14:58.762577 (+ 5043us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:14:58.762586 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:14:58.762601 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:14:58.763121 (+ 520us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:14:58.763731 (+ 610us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:14:58.763735 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:14:58.763738 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:14:58.763963 (+ 225us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:14:58.765138 (+ 1175us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:14:58.765145 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:14:58.765557 (+ 412us) client_negotiation.cc:769] Sending connection context
0928 02:14:58.765824 (+ 267us) client_negotiation.cc:240] Negotiation successful
0928 02:14:58.766282 (+ 458us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":637,"thread_start_us":387,"threads_started":1}
I20260928 02:14:58.768042 23358 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:14:58.742631 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.235.65:60259 (local address 127.6.235.126:46725)
0928 02:14:58.743329 (+ 698us) server_negotiation.cc:206] Beginning negotiation
0928 02:14:58.743348 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0928 02:14:58.745012 (+ 1664us) server_negotiation.cc:407] Connection header received
0928 02:14:58.746808 (+ 1796us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:14:58.746823 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:14:58.747220 (+ 397us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:14:58.747650 (+ 430us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:14:58.749416 (+ 1766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:14:58.750702 (+ 1286us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:14:58.752427 (+ 1725us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:14:58.753693 (+ 1266us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:14:58.753908 (+ 215us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:14:58.757873 (+ 3965us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:14:58.757886 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:14:58.757910 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:14:58.757977 (+ 67us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:14:58.762374 (+ 4397us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:14:58.763297 (+ 923us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:14:58.763309 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:14:58.763321 (+ 12us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:14:58.763578 (+ 257us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:14:58.764107 (+ 529us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:14:58.764115 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:14:58.764120 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:14:58.764981 (+ 861us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:14:58.765209 (+ 228us) server_negotiation.cc:1035] Waiting for connection context
0928 02:14:58.766137 (+ 928us) server_negotiation.cc:299] Negotiation successful
0928 02:14:58.766670 (+ 533us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":425,"thread_start_us":266,"threads_started":1}
I20260928 02:14:58.770879 23356 heartbeater.cc:347] Connected to a master server at 127.6.235.126:46725
I20260928 02:14:58.771224 23356 heartbeater.cc:464] Registering TS with master...
I20260928 02:14:58.771981 23356 heartbeater.cc:511] Master 127.6.235.126:46725 requested a full tablet report, sending...
I20260928 02:14:58.774345 23171 ts_manager.cc:194] Registered new tserver with Master: c8d284c745a3487e80f359a662875c9d (127.6.235.65:33947)
I20260928 02:14:58.775938 23171 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.235.65@KRBTEST.COM'} at 127.6.235.65:60259
W20260928 02:14:59.098508 23360 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:14:59.098870 23360 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:14:59.099026 23360 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:14:59.106688 23360 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:14:59.106858 23360 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:14:59.107028 23360 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.235.66
I20260928 02:14:59.117296 23360 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.235.84:46687
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790561432251112-7085-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790561432251112-7085-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790561432251112-7085-0/krb5kdc/kudu_127.6.235.66.keytab
--principal=kudu/127.6.235.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.235.66:46099
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790561432251112-7085-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.6.235.66
--webserver_port=43345
--tserver_master_addrs=127.6.235.126:46725
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.235.66
--log_dir=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790561432251112-7085-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 00:51:59 UTC on ccycloud.quasar-axgkwn.root.comops.site
build id 4676464
ASAN enabled
I20260928 02:14:59.119119 23360 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:14:59.120570 23360 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:14:59.135143 23365 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:14:59.135090 23366 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:14:59.135040 23368 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:14:59.136377 23360 server_base.cc:1034] running on GCE node
I20260928 02:14:59.137300 23360 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:14:59.138665 23360 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:14:59.139932 23360 hybrid_clock.cc:648] HybridClock initialized: now 1790561699139839 us; error 84 us; skew 500 ppm
I20260928 02:14:59.140337 23360 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 02:14:59 dist-test-slave-86gv krb5kdc[22587](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561699, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.235.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:14:59.144423 23360 init.cc:377] Logged in from keytab as kudu/127.6.235.66@KRBTEST.COM (short username kudu)
I20260928 02:14:59.146286 23360 webserver.cc:460] Webserver started at http://127.6.235.66:43345/ using document root <none> and password file <none>
I20260928 02:14:59.147122 23360 fs_manager.cc:359] Metadata directory not provided
I20260928 02:14:59.147310 23360 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:14:59.154114 23360 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.001s
I20260928 02:14:59.157773 23375 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:14:59.159654 23360 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260928 02:14:59.159884 23360 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790561432251112-7085-0/minicluster-data/ts-1/data,/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790561432251112-7085-0/minicluster-data/ts-1/wal
uuid: "6c165d86953345f6a45e9812e72c1b70"
format_stamp: "Formatted at 2026-09-28 02:14:36 on dist-test-slave-86gv"
server_key: "faed87a17ceb68956d872503da8c75b5"
server_key_iv: "ad63fd5a32bacd6faf8da18b56601cbe"
server_key_version: "encryptionkey@0"
I20260928 02:14:59.161015 23360 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790561432251112-7085-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790561432251112-7085-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790561432251112-7085-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:14:59.196069 23360 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:14:59.201493 23360 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:14:59.201807 23360 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:14:59.202890 23360 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:14:59.204401 23360 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:14:59.204523 23360 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:14:59.204650 23360 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:14:59.204734 23360 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:14:59.240586 23360 rpc_server.cc:307] RPC server started. Bound to: 127.6.235.66:46099
I20260928 02:14:59.240597 23487 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.235.66:46099 every 8 connection(s)
I20260928 02:14:59.242281 23360 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790561432251112-7085-0/minicluster-data/ts-1/data/info.pb
I20260928 02:14:59.243134 7085 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTlaUuc/build/asan/bin/kudu as pid 23360
I20260928 02:14:59.243930 7085 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTlaUuc/build/asan/bin/kudu
/tmp/dist-test-taskTlaUuc/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790561432251112-7085-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790561432251112-7085-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790561432251112-7085-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790561432251112-7085-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.235.67:37203
--local_ip_for_outbound_sockets=127.6.235.67
--tserver_master_addrs=127.6.235.126:46725
--webserver_port=34857
--webserver_interface=127.6.235.67
--keytab_file=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790561432251112-7085-0/krb5kdc/kudu_127.6.235.67.keytab
--principal=kudu/127.6.235.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.235.84:46687
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790561432251112-7085-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790561432251112-7085-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790561432251112-7085-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 28 02:14:59 dist-test-slave-86gv krb5kdc[22587](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561699, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.235.66@KRBTEST.COM for kudu/127.6.235.126@KRBTEST.COM
I20260928 02:14:59.267833 23358 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:14:59.245627 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.235.66:46973 (local address 127.6.235.126:46725)
0928 02:14:59.245875 (+ 248us) server_negotiation.cc:206] Beginning negotiation
0928 02:14:59.245883 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 02:14:59.247332 (+ 1449us) server_negotiation.cc:407] Connection header received
0928 02:14:59.250568 (+ 3236us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:14:59.250574 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:14:59.250702 (+ 128us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:14:59.250831 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:14:59.253409 (+ 2578us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:14:59.254343 (+ 934us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:14:59.256190 (+ 1847us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:14:59.256652 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:14:59.256805 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:14:59.261313 (+ 4508us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:14:59.261321 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:14:59.261325 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:14:59.261366 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:14:59.264188 (+ 2822us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:14:59.265227 (+ 1039us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:14:59.265232 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:14:59.265234 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:14:59.265395 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:14:59.265891 (+ 496us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:14:59.265901 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:14:59.265903 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:14:59.266234 (+ 331us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:14:59.266401 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0928 02:14:59.267182 (+ 781us) server_negotiation.cc:299] Negotiation successful
0928 02:14:59.267455 (+ 273us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260928 02:14:59.269135 23490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:14:59.246169 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.235.126:46725 (local address 127.6.235.66:46973)
0928 02:14:59.247100 (+ 931us) negotiation.cc:107] Waiting for socket to connect
0928 02:14:59.247161 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0928 02:14:59.250094 (+ 2933us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:14:59.251166 (+ 1072us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:14:59.251182 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:14:59.252066 (+ 884us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:14:59.253110 (+ 1044us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:14:59.253146 (+ 36us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:14:59.254569 (+ 1423us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:14:59.254577 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:14:59.255997 (+ 1420us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:14:59.256012 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:14:59.256956 (+ 944us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:14:59.256966 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:14:59.257246 (+ 280us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:14:59.258014 (+ 768us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:14:59.258040 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:14:59.260945 (+ 2905us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:14:59.264389 (+ 3444us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:14:59.264408 (+ 19us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:14:59.264430 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:14:59.265044 (+ 614us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:14:59.265519 (+ 475us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:14:59.265525 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:14:59.265528 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:14:59.265755 (+ 227us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:14:59.266375 (+ 620us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:14:59.266383 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:14:59.266790 (+ 407us) client_negotiation.cc:769] Sending connection context
0928 02:14:59.267089 (+ 299us) client_negotiation.cc:240] Negotiation successful
0928 02:14:59.267722 (+ 633us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":594,"thread_start_us":379,"threads_started":1}
I20260928 02:14:59.270970 23488 heartbeater.cc:347] Connected to a master server at 127.6.235.126:46725
I20260928 02:14:59.271324 23488 heartbeater.cc:464] Registering TS with master...
I20260928 02:14:59.272112 23488 heartbeater.cc:511] Master 127.6.235.126:46725 requested a full tablet report, sending...
I20260928 02:14:59.273823 23171 ts_manager.cc:194] Registered new tserver with Master: 6c165d86953345f6a45e9812e72c1b70 (127.6.235.66:46099)
I20260928 02:14:59.274971 23171 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.235.66@KRBTEST.COM'} at 127.6.235.66:46973
W20260928 02:14:59.600782 23491 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:14:59.601119 23491 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:14:59.601293 23491 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:14:59.608815 23491 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:14:59.608973 23491 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:14:59.609149 23491 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.235.67
I20260928 02:14:59.619501 23491 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.235.84:46687
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790561432251112-7085-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790561432251112-7085-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790561432251112-7085-0/krb5kdc/kudu_127.6.235.67.keytab
--principal=kudu/127.6.235.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.235.67:37203
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790561432251112-7085-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.6.235.67
--webserver_port=34857
--tserver_master_addrs=127.6.235.126:46725
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.235.67
--log_dir=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790561432251112-7085-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 00:51:59 UTC on ccycloud.quasar-axgkwn.root.comops.site
build id 4676464
ASAN enabled
I20260928 02:14:59.620813 23491 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:14:59.622166 23491 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:14:59.636696 23497 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:14:59.636839 23496 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:14:59.636677 23499 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:14:59.643143 23491 server_base.cc:1034] running on GCE node
I20260928 02:14:59.644057 23491 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:14:59.645299 23491 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:14:59.646541 23491 hybrid_clock.cc:648] HybridClock initialized: now 1790561699646491 us; error 49 us; skew 500 ppm
I20260928 02:14:59.646938 23491 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 02:14:59 dist-test-slave-86gv krb5kdc[22587](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561699, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.235.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:14:59.650437 23491 init.cc:377] Logged in from keytab as kudu/127.6.235.67@KRBTEST.COM (short username kudu)
I20260928 02:14:59.652240 23491 webserver.cc:460] Webserver started at http://127.6.235.67:34857/ using document root <none> and password file <none>
I20260928 02:14:59.653195 23491 fs_manager.cc:359] Metadata directory not provided
I20260928 02:14:59.653321 23491 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:14:59.659261 23491 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.004s sys 0.000s
I20260928 02:14:59.662431 23506 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:14:59.664053 23491 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260928 02:14:59.664247 23491 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790561432251112-7085-0/minicluster-data/ts-2/data,/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790561432251112-7085-0/minicluster-data/ts-2/wal
uuid: "613712f91b1348c29b808745f72d275a"
format_stamp: "Formatted at 2026-09-28 02:14:36 on dist-test-slave-86gv"
server_key: "0f80c683ff15b55d4fcfe676df3711df"
server_key_iv: "68eaf86367030440b44df4090297600d"
server_key_version: "encryptionkey@0"
I20260928 02:14:59.665448 23491 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790561432251112-7085-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790561432251112-7085-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790561432251112-7085-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:14:59.698413 23491 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:14:59.703199 23491 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:14:59.703503 23491 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:14:59.704618 23491 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:14:59.706162 23491 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:14:59.706251 23491 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:14:59.706400 23491 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:14:59.706467 23491 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:14:59.745743 23491 rpc_server.cc:307] RPC server started. Bound to: 127.6.235.67:37203
I20260928 02:14:59.745837 23618 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.235.67:37203 every 8 connection(s)
I20260928 02:14:59.747234 23491 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790561432251112-7085-0/minicluster-data/ts-2/data/info.pb
I20260928 02:14:59.754446 7085 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTlaUuc/build/asan/bin/kudu as pid 23491
Sep 28 02:14:59 dist-test-slave-86gv krb5kdc[22587](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561699, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.235.67@KRBTEST.COM for kudu/127.6.235.126@KRBTEST.COM
I20260928 02:14:59.770284 23358 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:14:59.750037 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.235.67:39787 (local address 127.6.235.126:46725)
0928 02:14:59.750250 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0928 02:14:59.750256 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 02:14:59.751585 (+ 1329us) server_negotiation.cc:407] Connection header received
0928 02:14:59.753583 (+ 1998us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:14:59.753588 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:14:59.753657 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:14:59.753755 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:14:59.755854 (+ 2099us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:14:59.756609 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:14:59.758401 (+ 1792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:14:59.759148 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:14:59.759291 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:14:59.763033 (+ 3742us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:14:59.763041 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:14:59.763046 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:14:59.763088 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:14:59.766439 (+ 3351us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:14:59.767504 (+ 1065us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:14:59.767509 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:14:59.767512 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:14:59.767687 (+ 175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:14:59.768245 (+ 558us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:14:59.768250 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:14:59.768258 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:14:59.768552 (+ 294us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:14:59.768726 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0928 02:14:59.769650 (+ 924us) server_negotiation.cc:299] Negotiation successful
0928 02:14:59.769876 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260928 02:14:59.771405 23621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:14:59.750591 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.235.126:46725 (local address 127.6.235.67:39787)
0928 02:14:59.751337 (+ 746us) negotiation.cc:107] Waiting for socket to connect
0928 02:14:59.751462 (+ 125us) client_negotiation.cc:174] Beginning negotiation
0928 02:14:59.753320 (+ 1858us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:14:59.754071 (+ 751us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:14:59.754079 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:14:59.754747 (+ 668us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:14:59.755566 (+ 819us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:14:59.755600 (+ 34us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:14:59.756775 (+ 1175us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:14:59.756783 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:14:59.758249 (+ 1466us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:14:59.758261 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:14:59.759484 (+ 1223us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:14:59.759493 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:14:59.759737 (+ 244us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:14:59.760501 (+ 764us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:14:59.760524 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:14:59.762822 (+ 2298us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:14:59.766603 (+ 3781us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:14:59.766614 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:14:59.766634 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:14:59.767302 (+ 668us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:14:59.767811 (+ 509us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:14:59.767817 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:14:59.767820 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:14:59.768117 (+ 297us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:14:59.768692 (+ 575us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:14:59.768701 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:14:59.769225 (+ 524us) client_negotiation.cc:769] Sending connection context
0928 02:14:59.769531 (+ 306us) client_negotiation.cc:240] Negotiation successful
0928 02:14:59.770121 (+ 590us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":489,"thread_start_us":288,"threads_started":1}
I20260928 02:14:59.772794 23622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:14:59.755342 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51884 (local address 127.6.235.126:46725)
0928 02:14:59.755872 (+ 530us) server_negotiation.cc:206] Beginning negotiation
0928 02:14:59.755877 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 02:14:59.757461 (+ 1584us) server_negotiation.cc:407] Connection header received
0928 02:14:59.757616 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:14:59.757620 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:14:59.757681 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:14:59.757786 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:14:59.759686 (+ 1900us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:14:59.760584 (+ 898us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:14:59.762018 (+ 1434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:14:59.762498 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:14:59.762626 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:14:59.766253 (+ 3627us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:14:59.766258 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:14:59.766262 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:14:59.766311 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:14:59.769715 (+ 3404us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:14:59.770670 (+ 955us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:14:59.770676 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:14:59.770679 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:14:59.770799 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:14:59.771208 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:14:59.771213 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:14:59.771217 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:14:59.771495 (+ 278us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:14:59.771753 (+ 258us) server_negotiation.cc:1035] Waiting for connection context
0928 02:14:59.772196 (+ 443us) server_negotiation.cc:299] Negotiation successful
0928 02:14:59.772437 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":409,"thread_start_us":309,"threads_started":1}
I20260928 02:14:59.773032 23619 heartbeater.cc:347] Connected to a master server at 127.6.235.126:46725
I20260928 02:14:59.773329 23619 heartbeater.cc:464] Registering TS with master...
I20260928 02:14:59.774231 23619 heartbeater.cc:511] Master 127.6.235.126:46725 requested a full tablet report, sending...
I20260928 02:14:59.775955 23171 ts_manager.cc:194] Registered new tserver with Master: 613712f91b1348c29b808745f72d275a (127.6.235.67:37203)
I20260928 02:14:59.776731 23171 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.235.67@KRBTEST.COM'} at 127.6.235.67:39787
I20260928 02:14:59.779219 23356 heartbeater.cc:503] Master 127.6.235.126:46725 was elected leader, sending a full tablet report...
I20260928 02:14:59.788050 7085 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 02:14:59.795832 23622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:14:59.789159 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51894 (local address 127.6.235.126:46725)
0928 02:14:59.789458 (+ 299us) server_negotiation.cc:206] Beginning negotiation
0928 02:14:59.789465 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 02:14:59.789801 (+ 336us) server_negotiation.cc:407] Connection header received
0928 02:14:59.790126 (+ 325us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:14:59.790131 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:14:59.790285 (+ 154us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:14:59.790497 (+ 212us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 02:14:59.791016 (+ 519us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:14:59.791795 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:14:59.793568 (+ 1773us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:14:59.794048 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:14:59.794199 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:14:59.794799 (+ 600us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 02:14:59.795046 (+ 247us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_AUTHENTICATION_TOKEN: Not authorized: token expired
0928 02:14:59.795488 (+ 442us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:51894: token expired
Metrics: {"server-negotiator.queue_time_us":116}
W20260928 02:14:59.795977 23622 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:51894: token expired
W20260928 02:14:59.796674 23093 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.6.235.126:46725)
I20260928 02:14:59.810489 23622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:14:59.797334 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51904 (local address 127.6.235.126:46725)
0928 02:14:59.797512 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0928 02:14:59.797516 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:14:59.797533 (+ 17us) server_negotiation.cc:407] Connection header received
0928 02:14:59.797620 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:14:59.797624 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:14:59.797684 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:14:59.797770 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:14:59.799184 (+ 1414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:14:59.799982 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:14:59.801540 (+ 1558us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:14:59.801974 (+ 434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:14:59.802076 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:14:59.805007 (+ 2931us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:14:59.805012 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:14:59.805015 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:14:59.805050 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:14:59.807704 (+ 2654us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:14:59.808595 (+ 891us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:14:59.808599 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:14:59.808602 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:14:59.808728 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:14:59.809147 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:14:59.809151 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:14:59.809153 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:14:59.809505 (+ 352us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:14:59.809693 (+ 188us) server_negotiation.cc:1035] Waiting for connection context
0928 02:14:59.810049 (+ 356us) server_negotiation.cc:299] Negotiation successful
0928 02:14:59.810226 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260928 02:14:59.814334 23171 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51904:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 02:14:59.822371 23171 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 02:14:59.850251 23629 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:14:59.836625 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.235.65:33947 (local address 127.0.0.1:36470)
0928 02:14:59.837558 (+ 933us) negotiation.cc:107] Waiting for socket to connect
0928 02:14:59.837613 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0928 02:14:59.837836 (+ 223us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:14:59.839338 (+ 1502us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:14:59.839350 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:14:59.839391 (+ 41us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:14:59.839951 (+ 560us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:14:59.839991 (+ 40us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:14:59.842682 (+ 2691us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:14:59.842692 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:14:59.845931 (+ 3239us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:14:59.845947 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:14:59.848851 (+ 2904us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:14:59.848862 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:14:59.849278 (+ 416us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:14:59.849371 (+ 93us) client_negotiation.cc:769] Sending connection context
0928 02:14:59.849526 (+ 155us) client_negotiation.cc:240] Negotiation successful
0928 02:14:59.849830 (+ 304us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":655,"thread_start_us":993,"threads_started":1}
I20260928 02:14:59.852085 23630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:14:59.836870 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36470 (local address 127.6.235.65:33947)
0928 02:14:59.838480 (+ 1610us) server_negotiation.cc:206] Beginning negotiation
0928 02:14:59.838489 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 02:14:59.838518 (+ 29us) server_negotiation.cc:407] Connection header received
0928 02:14:59.838620 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:14:59.838628 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:14:59.839041 (+ 413us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:14:59.840311 (+ 1270us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:14:59.840607 (+ 296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:14:59.842478 (+ 1871us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:14:59.846226 (+ 3748us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:14:59.848451 (+ 2225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:14:59.850848 (+ 2397us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:14:59.851201 (+ 353us) server_negotiation.cc:1035] Waiting for connection context
0928 02:14:59.851310 (+ 109us) server_negotiation.cc:299] Negotiation successful
0928 02:14:59.851608 (+ 298us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1435,"thread_start_us":1259,"threads_started":1}
I20260928 02:14:59.858023 23291 tablet_service.cc:1467] Processing CreateTablet for tablet 92a44f408aeb4bb283302518b7a88de8 (DEFAULT_TABLE table=test-table [id=04e32b0f05a1478aa551f468b401342e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:14:59.860045 23291 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 92a44f408aeb4bb283302518b7a88de8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:14:59.864686 23629 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:14:59.836859 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.235.67:37203 (local address 127.0.0.1:57548)
0928 02:14:59.850540 (+ 13681us) negotiation.cc:107] Waiting for socket to connect
0928 02:14:59.850560 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0928 02:14:59.850676 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:14:59.858675 (+ 7999us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:14:59.858684 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:14:59.858720 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:14:59.858940 (+ 220us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:14:59.858955 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:14:59.860624 (+ 1669us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:14:59.860632 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:14:59.862232 (+ 1600us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:14:59.862263 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:14:59.863728 (+ 1465us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:14:59.863734 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:14:59.863910 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:14:59.863975 (+ 65us) client_negotiation.cc:769] Sending connection context
0928 02:14:59.864115 (+ 140us) client_negotiation.cc:240] Negotiation successful
0928 02:14:59.864274 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13575,"thread_start_us":14721,"threads_started":1}
I20260928 02:14:59.864727 23632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:14:59.837256 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57548 (local address 127.6.235.67:37203)
0928 02:14:59.857932 (+ 20676us) server_negotiation.cc:206] Beginning negotiation
0928 02:14:59.857942 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 02:14:59.857988 (+ 46us) server_negotiation.cc:407] Connection header received
0928 02:14:59.858115 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:14:59.858123 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:14:59.858400 (+ 277us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:14:59.858643 (+ 243us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:14:59.859147 (+ 504us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:14:59.860489 (+ 1342us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:14:59.862468 (+ 1979us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:14:59.863547 (+ 1079us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:14:59.863707 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:14:59.863965 (+ 258us) server_negotiation.cc:1035] Waiting for connection context
0928 02:14:59.864172 (+ 207us) server_negotiation.cc:299] Negotiation successful
0928 02:14:59.864339 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20464,"thread_start_us":20304,"threads_started":1}
I20260928 02:14:59.867075 23631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:14:59.839489 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.235.66:46099 (local address 127.0.0.1:54564)
0928 02:14:59.851943 (+ 12454us) negotiation.cc:107] Waiting for socket to connect
0928 02:14:59.851971 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0928 02:14:59.852133 (+ 162us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:14:59.858995 (+ 6862us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:14:59.859001 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:14:59.859031 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:14:59.859460 (+ 429us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:14:59.859475 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:14:59.861610 (+ 2135us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:14:59.861616 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:14:59.864172 (+ 2556us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:14:59.864191 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:14:59.866334 (+ 2143us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:14:59.866340 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:14:59.866590 (+ 250us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:14:59.866616 (+ 26us) client_negotiation.cc:769] Sending connection context
0928 02:14:59.866724 (+ 108us) client_negotiation.cc:240] Negotiation successful
0928 02:14:59.866829 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12280,"thread_start_us":17672,"threads_started":1}
I20260928 02:14:59.867375 23634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:14:59.839817 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54564 (local address 127.6.235.66:46099)
0928 02:14:59.858230 (+ 18413us) server_negotiation.cc:206] Beginning negotiation
0928 02:14:59.858238 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 02:14:59.858264 (+ 26us) server_negotiation.cc:407] Connection header received
0928 02:14:59.858359 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:14:59.858367 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:14:59.858594 (+ 227us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:14:59.858781 (+ 187us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:14:59.859657 (+ 876us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:14:59.861384 (+ 1727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:14:59.864406 (+ 3022us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:14:59.866084 (+ 1678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:14:59.866255 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:14:59.866494 (+ 239us) server_negotiation.cc:1035] Waiting for connection context
0928 02:14:59.866821 (+ 327us) server_negotiation.cc:299] Negotiation successful
0928 02:14:59.867011 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18262,"thread_start_us":18113,"threads_started":1}
I20260928 02:14:59.869750 23554 tablet_service.cc:1467] Processing CreateTablet for tablet 92a44f408aeb4bb283302518b7a88de8 (DEFAULT_TABLE table=test-table [id=04e32b0f05a1478aa551f468b401342e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:14:59.871361 23554 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 92a44f408aeb4bb283302518b7a88de8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:14:59.871502 23423 tablet_service.cc:1467] Processing CreateTablet for tablet 92a44f408aeb4bb283302518b7a88de8 (DEFAULT_TABLE table=test-table [id=04e32b0f05a1478aa551f468b401342e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:14:59.873044 23423 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 92a44f408aeb4bb283302518b7a88de8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:14:59.880350 23635 tablet_bootstrap.cc:492] T 92a44f408aeb4bb283302518b7a88de8 P c8d284c745a3487e80f359a662875c9d: Bootstrap starting.
I20260928 02:14:59.884703 23635 tablet_bootstrap.cc:654] T 92a44f408aeb4bb283302518b7a88de8 P c8d284c745a3487e80f359a662875c9d: Neither blocks nor log segments found. Creating new log.
I20260928 02:14:59.886358 23635 log.cc:824] T 92a44f408aeb4bb283302518b7a88de8 P c8d284c745a3487e80f359a662875c9d: Log is configured to *not* fsync() on all Append() calls
I20260928 02:14:59.887708 23636 tablet_bootstrap.cc:492] T 92a44f408aeb4bb283302518b7a88de8 P 6c165d86953345f6a45e9812e72c1b70: Bootstrap starting.
I20260928 02:14:59.889554 23638 tablet_bootstrap.cc:492] T 92a44f408aeb4bb283302518b7a88de8 P 613712f91b1348c29b808745f72d275a: Bootstrap starting.
I20260928 02:14:59.892984 23636 tablet_bootstrap.cc:654] T 92a44f408aeb4bb283302518b7a88de8 P 6c165d86953345f6a45e9812e72c1b70: Neither blocks nor log segments found. Creating new log.
I20260928 02:14:59.893309 23638 tablet_bootstrap.cc:654] T 92a44f408aeb4bb283302518b7a88de8 P 613712f91b1348c29b808745f72d275a: Neither blocks nor log segments found. Creating new log.
I20260928 02:14:59.893596 23635 tablet_bootstrap.cc:492] T 92a44f408aeb4bb283302518b7a88de8 P c8d284c745a3487e80f359a662875c9d: No bootstrap required, opened a new log
I20260928 02:14:59.893893 23635 ts_tablet_manager.cc:1397] T 92a44f408aeb4bb283302518b7a88de8 P c8d284c745a3487e80f359a662875c9d: Time spent bootstrapping tablet: real 0.014s user 0.003s sys 0.008s
I20260928 02:14:59.894379 23636 log.cc:824] T 92a44f408aeb4bb283302518b7a88de8 P 6c165d86953345f6a45e9812e72c1b70: Log is configured to *not* fsync() on all Append() calls
I20260928 02:14:59.894513 23638 log.cc:824] T 92a44f408aeb4bb283302518b7a88de8 P 613712f91b1348c29b808745f72d275a: Log is configured to *not* fsync() on all Append() calls
I20260928 02:14:59.897646 23636 tablet_bootstrap.cc:492] T 92a44f408aeb4bb283302518b7a88de8 P 6c165d86953345f6a45e9812e72c1b70: No bootstrap required, opened a new log
I20260928 02:14:59.897943 23636 ts_tablet_manager.cc:1397] T 92a44f408aeb4bb283302518b7a88de8 P 6c165d86953345f6a45e9812e72c1b70: Time spent bootstrapping tablet: real 0.010s user 0.004s sys 0.003s
I20260928 02:14:59.898083 23638 tablet_bootstrap.cc:492] T 92a44f408aeb4bb283302518b7a88de8 P 613712f91b1348c29b808745f72d275a: No bootstrap required, opened a new log
I20260928 02:14:59.898366 23638 ts_tablet_manager.cc:1397] T 92a44f408aeb4bb283302518b7a88de8 P 613712f91b1348c29b808745f72d275a: Time spent bootstrapping tablet: real 0.009s user 0.007s sys 0.000s
I20260928 02:14:59.905834 23636 raft_consensus.cc:348] T 92a44f408aeb4bb283302518b7a88de8 P 6c165d86953345f6a45e9812e72c1b70 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8d284c745a3487e80f359a662875c9d" member_type: VOTER last_known_addr { host: "127.6.235.65" port: 33947 } } peers { permanent_uuid: "613712f91b1348c29b808745f72d275a" member_type: VOTER last_known_addr { host: "127.6.235.67" port: 37203 } } peers { permanent_uuid: "6c165d86953345f6a45e9812e72c1b70" member_type: VOTER last_known_addr { host: "127.6.235.66" port: 46099 } }
I20260928 02:14:59.906252 23636 raft_consensus.cc:374] T 92a44f408aeb4bb283302518b7a88de8 P 6c165d86953345f6a45e9812e72c1b70 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:14:59.906379 23636 raft_consensus.cc:729] T 92a44f408aeb4bb283302518b7a88de8 P 6c165d86953345f6a45e9812e72c1b70 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6c165d86953345f6a45e9812e72c1b70, State: Initialized, Role: FOLLOWER
I20260928 02:14:59.906042 23635 raft_consensus.cc:348] T 92a44f408aeb4bb283302518b7a88de8 P c8d284c745a3487e80f359a662875c9d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8d284c745a3487e80f359a662875c9d" member_type: VOTER last_known_addr { host: "127.6.235.65" port: 33947 } } peers { permanent_uuid: "613712f91b1348c29b808745f72d275a" member_type: VOTER last_known_addr { host: "127.6.235.67" port: 37203 } } peers { permanent_uuid: "6c165d86953345f6a45e9812e72c1b70" member_type: VOTER last_known_addr { host: "127.6.235.66" port: 46099 } }
I20260928 02:14:59.906602 23635 raft_consensus.cc:374] T 92a44f408aeb4bb283302518b7a88de8 P c8d284c745a3487e80f359a662875c9d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:14:59.906781 23635 raft_consensus.cc:729] T 92a44f408aeb4bb283302518b7a88de8 P c8d284c745a3487e80f359a662875c9d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c8d284c745a3487e80f359a662875c9d, State: Initialized, Role: FOLLOWER
I20260928 02:14:59.907095 23636 consensus_queue.cc:260] T 92a44f408aeb4bb283302518b7a88de8 P 6c165d86953345f6a45e9812e72c1b70 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8d284c745a3487e80f359a662875c9d" member_type: VOTER last_known_addr { host: "127.6.235.65" port: 33947 } } peers { permanent_uuid: "613712f91b1348c29b808745f72d275a" member_type: VOTER last_known_addr { host: "127.6.235.67" port: 37203 } } peers { permanent_uuid: "6c165d86953345f6a45e9812e72c1b70" member_type: VOTER last_known_addr { host: "127.6.235.66" port: 46099 } }
I20260928 02:14:59.907447 23635 consensus_queue.cc:260] T 92a44f408aeb4bb283302518b7a88de8 P c8d284c745a3487e80f359a662875c9d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8d284c745a3487e80f359a662875c9d" member_type: VOTER last_known_addr { host: "127.6.235.65" port: 33947 } } peers { permanent_uuid: "613712f91b1348c29b808745f72d275a" member_type: VOTER last_known_addr { host: "127.6.235.67" port: 37203 } } peers { permanent_uuid: "6c165d86953345f6a45e9812e72c1b70" member_type: VOTER last_known_addr { host: "127.6.235.66" port: 46099 } }
I20260928 02:14:59.908320 23488 heartbeater.cc:503] Master 127.6.235.126:46725 was elected leader, sending a full tablet report...
I20260928 02:14:59.909111 23636 ts_tablet_manager.cc:1428] T 92a44f408aeb4bb283302518b7a88de8 P 6c165d86953345f6a45e9812e72c1b70: Time spent starting tablet: real 0.011s user 0.011s sys 0.001s
I20260928 02:14:59.911038 23635 ts_tablet_manager.cc:1428] T 92a44f408aeb4bb283302518b7a88de8 P c8d284c745a3487e80f359a662875c9d: Time spent starting tablet: real 0.017s user 0.014s sys 0.003s
I20260928 02:14:59.910753 23638 raft_consensus.cc:348] T 92a44f408aeb4bb283302518b7a88de8 P 613712f91b1348c29b808745f72d275a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8d284c745a3487e80f359a662875c9d" member_type: VOTER last_known_addr { host: "127.6.235.65" port: 33947 } } peers { permanent_uuid: "613712f91b1348c29b808745f72d275a" member_type: VOTER last_known_addr { host: "127.6.235.67" port: 37203 } } peers { permanent_uuid: "6c165d86953345f6a45e9812e72c1b70" member_type: VOTER last_known_addr { host: "127.6.235.66" port: 46099 } }
I20260928 02:14:59.911339 23638 raft_consensus.cc:374] T 92a44f408aeb4bb283302518b7a88de8 P 613712f91b1348c29b808745f72d275a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:14:59.911495 23638 raft_consensus.cc:729] T 92a44f408aeb4bb283302518b7a88de8 P 613712f91b1348c29b808745f72d275a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 613712f91b1348c29b808745f72d275a, State: Initialized, Role: FOLLOWER
I20260928 02:14:59.912212 23638 consensus_queue.cc:260] T 92a44f408aeb4bb283302518b7a88de8 P 613712f91b1348c29b808745f72d275a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8d284c745a3487e80f359a662875c9d" member_type: VOTER last_known_addr { host: "127.6.235.65" port: 33947 } } peers { permanent_uuid: "613712f91b1348c29b808745f72d275a" member_type: VOTER last_known_addr { host: "127.6.235.67" port: 37203 } } peers { permanent_uuid: "6c165d86953345f6a45e9812e72c1b70" member_type: VOTER last_known_addr { host: "127.6.235.66" port: 46099 } }
I20260928 02:14:59.913379 23619 heartbeater.cc:503] Master 127.6.235.126:46725 was elected leader, sending a full tablet report...
I20260928 02:14:59.914146 23638 ts_tablet_manager.cc:1428] T 92a44f408aeb4bb283302518b7a88de8 P 613712f91b1348c29b808745f72d275a: Time spent starting tablet: real 0.016s user 0.016s sys 0.000s
W20260928 02:14:59.991914 23357 tablet.cc:2369] T 92a44f408aeb4bb283302518b7a88de8 P c8d284c745a3487e80f359a662875c9d: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 02:14:59.994643 23489 tablet.cc:2369] T 92a44f408aeb4bb283302518b7a88de8 P 6c165d86953345f6a45e9812e72c1b70: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 02:14:59.998901 23620 tablet.cc:2369] T 92a44f408aeb4bb283302518b7a88de8 P 613712f91b1348c29b808745f72d275a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 02:15:00.049974 23641 raft_consensus.cc:482] T 92a44f408aeb4bb283302518b7a88de8 P 6c165d86953345f6a45e9812e72c1b70 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 02:15:00.050339 23641 raft_consensus.cc:504] T 92a44f408aeb4bb283302518b7a88de8 P 6c165d86953345f6a45e9812e72c1b70 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8d284c745a3487e80f359a662875c9d" member_type: VOTER last_known_addr { host: "127.6.235.65" port: 33947 } } peers { permanent_uuid: "613712f91b1348c29b808745f72d275a" member_type: VOTER last_known_addr { host: "127.6.235.67" port: 37203 } } peers { permanent_uuid: "6c165d86953345f6a45e9812e72c1b70" member_type: VOTER last_known_addr { host: "127.6.235.66" port: 46099 } }
I20260928 02:15:00.052284 23641 leader_election.cc:288] T 92a44f408aeb4bb283302518b7a88de8 P 6c165d86953345f6a45e9812e72c1b70 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c8d284c745a3487e80f359a662875c9d (127.6.235.65:33947), 613712f91b1348c29b808745f72d275a (127.6.235.67:37203)
I20260928 02:15:00.059684 23630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:15:00.052751 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.235.66:57339 (local address 127.6.235.65:33947)
0928 02:15:00.053056 (+ 305us) server_negotiation.cc:206] Beginning negotiation
0928 02:15:00.053066 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 02:15:00.054351 (+ 1285us) server_negotiation.cc:407] Connection header received
0928 02:15:00.054562 (+ 211us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:15:00.054568 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:15:00.054638 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:15:00.054735 (+ 97us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:15:00.055286 (+ 551us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:15:00.055994 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:15:00.057814 (+ 1820us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:15:00.058613 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:15:00.058820 (+ 207us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:15:00.058918 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0928 02:15:00.059230 (+ 312us) server_negotiation.cc:299] Negotiation successful
0928 02:15:00.059362 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260928 02:15:00.059751 23632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:15:00.053195 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.235.66:60025 (local address 127.6.235.67:37203)
0928 02:15:00.053475 (+ 280us) server_negotiation.cc:206] Beginning negotiation
0928 02:15:00.053485 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 02:15:00.053842 (+ 357us) server_negotiation.cc:407] Connection header received
0928 02:15:00.054106 (+ 264us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:15:00.054112 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:15:00.054183 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:15:00.054291 (+ 108us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:15:00.054863 (+ 572us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:15:00.055672 (+ 809us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:15:00.057749 (+ 2077us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:15:00.058645 (+ 896us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:15:00.058867 (+ 222us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:15:00.058965 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0928 02:15:00.059276 (+ 311us) server_negotiation.cc:299] Negotiation successful
0928 02:15:00.059426 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87}
I20260928 02:15:00.059839 23645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:15:00.052835 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.235.65:33947 (local address 127.6.235.66:57339)
0928 02:15:00.054234 (+ 1399us) negotiation.cc:107] Waiting for socket to connect
0928 02:15:00.054254 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0928 02:15:00.054415 (+ 161us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:15:00.054750 (+ 335us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:15:00.054754 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:15:00.054778 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:15:00.055133 (+ 355us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:15:00.055142 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:15:00.056138 (+ 996us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:15:00.056143 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:15:00.057650 (+ 1507us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:15:00.057663 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:15:00.058864 (+ 1201us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:15:00.058871 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:15:00.059060 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:15:00.059090 (+ 30us) client_negotiation.cc:769] Sending connection context
0928 02:15:00.059200 (+ 110us) client_negotiation.cc:240] Negotiation successful
0928 02:15:00.059344 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1295,"thread_start_us":1026,"threads_started":1}
I20260928 02:15:00.059959 23644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:15:00.052647 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.235.67:37203 (local address 127.6.235.66:60025)
0928 02:15:00.053711 (+ 1064us) negotiation.cc:107] Waiting for socket to connect
0928 02:15:00.053728 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0928 02:15:00.053937 (+ 209us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:15:00.054309 (+ 372us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:15:00.054314 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:15:00.054341 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:15:00.054708 (+ 367us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:15:00.054718 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:15:00.055823 (+ 1105us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:15:00.055831 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:15:00.057558 (+ 1727us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:15:00.057572 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:15:00.058901 (+ 1329us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:15:00.058908 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:15:00.059101 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:15:00.059127 (+ 26us) client_negotiation.cc:769] Sending connection context
0928 02:15:00.059241 (+ 114us) client_negotiation.cc:240] Negotiation successful
0928 02:15:00.059397 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":928,"mutex_wait_us":4,"thread_start_us":706,"threads_started":1}
I20260928 02:15:00.061254 23574 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "92a44f408aeb4bb283302518b7a88de8" candidate_uuid: "6c165d86953345f6a45e9812e72c1b70" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "613712f91b1348c29b808745f72d275a" is_pre_election: true
I20260928 02:15:00.061254 23311 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "92a44f408aeb4bb283302518b7a88de8" candidate_uuid: "6c165d86953345f6a45e9812e72c1b70" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c8d284c745a3487e80f359a662875c9d" is_pre_election: true
I20260928 02:15:00.061894 23574 raft_consensus.cc:2454] T 92a44f408aeb4bb283302518b7a88de8 P 613712f91b1348c29b808745f72d275a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6c165d86953345f6a45e9812e72c1b70 in term 0.
I20260928 02:15:00.061898 23311 raft_consensus.cc:2454] T 92a44f408aeb4bb283302518b7a88de8 P c8d284c745a3487e80f359a662875c9d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6c165d86953345f6a45e9812e72c1b70 in term 0.
I20260928 02:15:00.062752 23376 leader_election.cc:302] T 92a44f408aeb4bb283302518b7a88de8 P 6c165d86953345f6a45e9812e72c1b70 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 613712f91b1348c29b808745f72d275a, 6c165d86953345f6a45e9812e72c1b70, c8d284c745a3487e80f359a662875c9d; no voters:
I20260928 02:15:00.063321 23641 raft_consensus.cc:2781] T 92a44f408aeb4bb283302518b7a88de8 P 6c165d86953345f6a45e9812e72c1b70 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 02:15:00.063591 23641 raft_consensus.cc:482] T 92a44f408aeb4bb283302518b7a88de8 P 6c165d86953345f6a45e9812e72c1b70 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 02:15:00.063717 23641 raft_consensus.cc:3037] T 92a44f408aeb4bb283302518b7a88de8 P 6c165d86953345f6a45e9812e72c1b70 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:15:00.066385 23641 raft_consensus.cc:504] T 92a44f408aeb4bb283302518b7a88de8 P 6c165d86953345f6a45e9812e72c1b70 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8d284c745a3487e80f359a662875c9d" member_type: VOTER last_known_addr { host: "127.6.235.65" port: 33947 } } peers { permanent_uuid: "613712f91b1348c29b808745f72d275a" member_type: VOTER last_known_addr { host: "127.6.235.67" port: 37203 } } peers { permanent_uuid: "6c165d86953345f6a45e9812e72c1b70" member_type: VOTER last_known_addr { host: "127.6.235.66" port: 46099 } }
I20260928 02:15:00.067078 23641 leader_election.cc:288] T 92a44f408aeb4bb283302518b7a88de8 P 6c165d86953345f6a45e9812e72c1b70 [CANDIDATE]: Term 1 election: Requested vote from peers c8d284c745a3487e80f359a662875c9d (127.6.235.65:33947), 613712f91b1348c29b808745f72d275a (127.6.235.67:37203)
I20260928 02:15:00.067713 23574 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "92a44f408aeb4bb283302518b7a88de8" candidate_uuid: "6c165d86953345f6a45e9812e72c1b70" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "613712f91b1348c29b808745f72d275a"
I20260928 02:15:00.067776 23311 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "92a44f408aeb4bb283302518b7a88de8" candidate_uuid: "6c165d86953345f6a45e9812e72c1b70" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c8d284c745a3487e80f359a662875c9d"
I20260928 02:15:00.067992 23574 raft_consensus.cc:3037] T 92a44f408aeb4bb283302518b7a88de8 P 613712f91b1348c29b808745f72d275a [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:15:00.068075 23311 raft_consensus.cc:3037] T 92a44f408aeb4bb283302518b7a88de8 P c8d284c745a3487e80f359a662875c9d [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:15:00.070684 23311 raft_consensus.cc:2454] T 92a44f408aeb4bb283302518b7a88de8 P c8d284c745a3487e80f359a662875c9d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6c165d86953345f6a45e9812e72c1b70 in term 1.
I20260928 02:15:00.070748 23574 raft_consensus.cc:2454] T 92a44f408aeb4bb283302518b7a88de8 P 613712f91b1348c29b808745f72d275a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6c165d86953345f6a45e9812e72c1b70 in term 1.
I20260928 02:15:00.071223 23377 leader_election.cc:302] T 92a44f408aeb4bb283302518b7a88de8 P 6c165d86953345f6a45e9812e72c1b70 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6c165d86953345f6a45e9812e72c1b70, c8d284c745a3487e80f359a662875c9d; no voters:
I20260928 02:15:00.071594 23641 raft_consensus.cc:2781] T 92a44f408aeb4bb283302518b7a88de8 P 6c165d86953345f6a45e9812e72c1b70 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:15:00.072086 23641 raft_consensus.cc:686] T 92a44f408aeb4bb283302518b7a88de8 P 6c165d86953345f6a45e9812e72c1b70 [term 1 LEADER]: Becoming Leader. State: Replica: 6c165d86953345f6a45e9812e72c1b70, State: Running, Role: LEADER
I20260928 02:15:00.072610 23641 consensus_queue.cc:237] T 92a44f408aeb4bb283302518b7a88de8 P 6c165d86953345f6a45e9812e72c1b70 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8d284c745a3487e80f359a662875c9d" member_type: VOTER last_known_addr { host: "127.6.235.65" port: 33947 } } peers { permanent_uuid: "613712f91b1348c29b808745f72d275a" member_type: VOTER last_known_addr { host: "127.6.235.67" port: 37203 } } peers { permanent_uuid: "6c165d86953345f6a45e9812e72c1b70" member_type: VOTER last_known_addr { host: "127.6.235.66" port: 46099 } }
I20260928 02:15:00.078413 23169 catalog_manager.cc:5495] T 92a44f408aeb4bb283302518b7a88de8 P 6c165d86953345f6a45e9812e72c1b70 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6c165d86953345f6a45e9812e72c1b70 (127.6.235.66). New cstate: current_term: 1 leader_uuid: "6c165d86953345f6a45e9812e72c1b70" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8d284c745a3487e80f359a662875c9d" member_type: VOTER last_known_addr { host: "127.6.235.65" port: 33947 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "613712f91b1348c29b808745f72d275a" member_type: VOTER last_known_addr { host: "127.6.235.67" port: 37203 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6c165d86953345f6a45e9812e72c1b70" member_type: VOTER last_known_addr { host: "127.6.235.66" port: 46099 } health_report { overall_health: HEALTHY } } }
I20260928 02:15:00.150816 23634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:15:00.144868 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54568 (local address 127.6.235.66:46099)
0928 02:15:00.145099 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0928 02:15:00.145106 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 02:15:00.145172 (+ 66us) server_negotiation.cc:407] Connection header received
0928 02:15:00.145450 (+ 278us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:15:00.145454 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:15:00.145535 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:15:00.145695 (+ 160us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 02:15:00.146093 (+ 398us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:15:00.146912 (+ 819us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:15:00.148458 (+ 1546us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:15:00.148876 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:15:00.149094 (+ 218us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:15:00.149622 (+ 528us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 02:15:00.150171 (+ 549us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 02:15:00.150364 (+ 193us) server_negotiation.cc:1035] Waiting for connection context
0928 02:15:00.150431 (+ 67us) server_negotiation.cc:299] Negotiation successful
0928 02:15:00.150523 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
I20260928 02:15:00.159914 23574 raft_consensus.cc:1261] T 92a44f408aeb4bb283302518b7a88de8 P 613712f91b1348c29b808745f72d275a [term 1 FOLLOWER]: Refusing update from remote peer 6c165d86953345f6a45e9812e72c1b70: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 02:15:00.159915 23311 raft_consensus.cc:1261] T 92a44f408aeb4bb283302518b7a88de8 P c8d284c745a3487e80f359a662875c9d [term 1 FOLLOWER]: Refusing update from remote peer 6c165d86953345f6a45e9812e72c1b70: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 02:15:00.160948 23646 consensus_queue.cc:1035] T 92a44f408aeb4bb283302518b7a88de8 P 6c165d86953345f6a45e9812e72c1b70 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c8d284c745a3487e80f359a662875c9d" member_type: VOTER last_known_addr { host: "127.6.235.65" port: 33947 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 02:15:00.161325 23641 consensus_queue.cc:1035] T 92a44f408aeb4bb283302518b7a88de8 P 6c165d86953345f6a45e9812e72c1b70 [LEADER]: Connected to new peer: Peer: permanent_uuid: "613712f91b1348c29b808745f72d275a" member_type: VOTER last_known_addr { host: "127.6.235.67" port: 37203 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 02:15:00.181567 23651 mvcc.cc:205] Tried to move back new op lower bound from 7334140723844358144 to 7334140723501707264. Current Snapshot: MvccSnapshot[applied={T|T < 7334140723844358144 or (T in {7334140723844358144})}]
I20260928 02:15:00.193900 23169 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51904:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 02:15:00.194242 23169 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51904:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 02:15:00.198462 23169 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 2bc05ba1194e4a02b77ee3f8d3ef5771: Sending DeleteTablet for 3 replicas of tablet 92a44f408aeb4bb283302518b7a88de8
I20260928 02:15:00.199831 23554 tablet_service.cc:1514] Processing DeleteTablet for tablet 92a44f408aeb4bb283302518b7a88de8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:15:00 UTC) from {username='kudu', principal='kudu/127.6.235.126@KRBTEST.COM'} at 127.0.0.1:57548
I20260928 02:15:00.199918 23291 tablet_service.cc:1514] Processing DeleteTablet for tablet 92a44f408aeb4bb283302518b7a88de8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:15:00 UTC) from {username='kudu', principal='kudu/127.6.235.126@KRBTEST.COM'} at 127.0.0.1:36470
I20260928 02:15:00.199918 23423 tablet_service.cc:1514] Processing DeleteTablet for tablet 92a44f408aeb4bb283302518b7a88de8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:15:00 UTC) from {username='kudu', principal='kudu/127.6.235.126@KRBTEST.COM'} at 127.0.0.1:54564
I20260928 02:15:00.200595 7085 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskTlaUuc/build/asan/bin/kudu pid 23208
I20260928 02:15:00.201756 23658 tablet_replica.cc:330] T 92a44f408aeb4bb283302518b7a88de8 P 613712f91b1348c29b808745f72d275a: stopping tablet replica
I20260928 02:15:00.202226 23658 raft_consensus.cc:2229] T 92a44f408aeb4bb283302518b7a88de8 P 613712f91b1348c29b808745f72d275a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 02:15:00.202759 23658 raft_consensus.cc:2258] T 92a44f408aeb4bb283302518b7a88de8 P 613712f91b1348c29b808745f72d275a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 02:15:00.203554 23660 tablet_replica.cc:330] T 92a44f408aeb4bb283302518b7a88de8 P c8d284c745a3487e80f359a662875c9d: stopping tablet replica
I20260928 02:15:00.203970 23660 raft_consensus.cc:2229] T 92a44f408aeb4bb283302518b7a88de8 P c8d284c745a3487e80f359a662875c9d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 02:15:00.204509 23660 raft_consensus.cc:2258] T 92a44f408aeb4bb283302518b7a88de8 P c8d284c745a3487e80f359a662875c9d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 02:15:00.205343 23659 tablet_replica.cc:330] T 92a44f408aeb4bb283302518b7a88de8 P 6c165d86953345f6a45e9812e72c1b70: stopping tablet replica
I20260928 02:15:00.207180 23658 ts_tablet_manager.cc:1905] T 92a44f408aeb4bb283302518b7a88de8 P 613712f91b1348c29b808745f72d275a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 02:15:00.207222 23660 ts_tablet_manager.cc:1905] T 92a44f408aeb4bb283302518b7a88de8 P c8d284c745a3487e80f359a662875c9d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 02:15:00.209334 23659 raft_consensus.cc:2229] T 92a44f408aeb4bb283302518b7a88de8 P 6c165d86953345f6a45e9812e72c1b70 [term 1 LEADER]: Raft consensus shutting down.
I20260928 02:15:00.210134 23659 raft_consensus.cc:2258] T 92a44f408aeb4bb283302518b7a88de8 P 6c165d86953345f6a45e9812e72c1b70 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 02:15:00.214720 23658 ts_tablet_manager.cc:1918] T 92a44f408aeb4bb283302518b7a88de8 P 613712f91b1348c29b808745f72d275a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 02:15:00.215022 23658 log.cc:1196] T 92a44f408aeb4bb283302518b7a88de8 P 613712f91b1348c29b808745f72d275a: Deleting WAL directory at /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790561432251112-7085-0/minicluster-data/ts-2/wal/wals/92a44f408aeb4bb283302518b7a88de8
I20260928 02:15:00.215718 23658 ts_tablet_manager.cc:1939] T 92a44f408aeb4bb283302518b7a88de8 P 613712f91b1348c29b808745f72d275a: Deleting consensus metadata
I20260928 02:15:00.216008 23660 ts_tablet_manager.cc:1918] T 92a44f408aeb4bb283302518b7a88de8 P c8d284c745a3487e80f359a662875c9d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 02:15:00.218007 23155 catalog_manager.cc:4841] TS 613712f91b1348c29b808745f72d275a (127.6.235.67:37203): tablet 92a44f408aeb4bb283302518b7a88de8 (table test-table [id=04e32b0f05a1478aa551f468b401342e]) successfully deleted
I20260928 02:15:00.218597 23660 log.cc:1196] T 92a44f408aeb4bb283302518b7a88de8 P c8d284c745a3487e80f359a662875c9d: Deleting WAL directory at /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790561432251112-7085-0/minicluster-data/ts-0/wal/wals/92a44f408aeb4bb283302518b7a88de8
I20260928 02:15:00.218691 23659 ts_tablet_manager.cc:1905] T 92a44f408aeb4bb283302518b7a88de8 P 6c165d86953345f6a45e9812e72c1b70: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 02:15:00.219269 23660 ts_tablet_manager.cc:1939] T 92a44f408aeb4bb283302518b7a88de8 P c8d284c745a3487e80f359a662875c9d: Deleting consensus metadata
I20260928 02:15:00.221380 23156 catalog_manager.cc:4841] TS c8d284c745a3487e80f359a662875c9d (127.6.235.65:33947): tablet 92a44f408aeb4bb283302518b7a88de8 (table test-table [id=04e32b0f05a1478aa551f468b401342e]) successfully deleted
I20260928 02:15:00.223107 23630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:15:00.202707 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36482 (local address 127.6.235.65:33947)
0928 02:15:00.202932 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0928 02:15:00.202941 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 02:15:00.202962 (+ 21us) server_negotiation.cc:407] Connection header received
0928 02:15:00.203038 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:15:00.203046 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:15:00.203152 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:15:00.203289 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:15:00.205930 (+ 2641us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:15:00.207161 (+ 1231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:15:00.209222 (+ 2061us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:15:00.209839 (+ 617us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:15:00.209967 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:15:00.213860 (+ 3893us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:15:00.213874 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:15:00.213898 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:15:00.213955 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:15:00.218366 (+ 4411us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:15:00.219449 (+ 1083us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:15:00.219463 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:15:00.219466 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:15:00.219609 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:15:00.221221 (+ 1612us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:15:00.221228 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:15:00.221230 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:15:00.221747 (+ 517us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:15:00.222069 (+ 322us) server_negotiation.cc:1035] Waiting for connection context
0928 02:15:00.222386 (+ 317us) server_negotiation.cc:299] Negotiation successful
0928 02:15:00.222625 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260928 02:15:00.223765 23351 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 02:15:00.226631 23659 ts_tablet_manager.cc:1918] T 92a44f408aeb4bb283302518b7a88de8 P 6c165d86953345f6a45e9812e72c1b70: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 02:15:00.226888 23659 log.cc:1196] T 92a44f408aeb4bb283302518b7a88de8 P 6c165d86953345f6a45e9812e72c1b70: Deleting WAL directory at /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790561432251112-7085-0/minicluster-data/ts-1/wal/wals/92a44f408aeb4bb283302518b7a88de8
I20260928 02:15:00.227599 23659 ts_tablet_manager.cc:1939] T 92a44f408aeb4bb283302518b7a88de8 P 6c165d86953345f6a45e9812e72c1b70: Deleting consensus metadata
I20260928 02:15:00.229643 23158 catalog_manager.cc:4841] TS 6c165d86953345f6a45e9812e72c1b70 (127.6.235.66:46099): tablet 92a44f408aeb4bb283302518b7a88de8 (table test-table [id=04e32b0f05a1478aa551f468b401342e]) successfully deleted
I20260928 02:15:00.336321 7085 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTlaUuc/build/asan/bin/kudu with pid 23208
I20260928 02:15:00.356607 7085 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskTlaUuc/build/asan/bin/kudu pid 23360
I20260928 02:15:00.374738 23634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:15:00.357487 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54580 (local address 127.6.235.66:46099)
0928 02:15:00.357704 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0928 02:15:00.357712 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 02:15:00.357837 (+ 125us) server_negotiation.cc:407] Connection header received
0928 02:15:00.358102 (+ 265us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:15:00.358106 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:15:00.358192 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:15:00.358280 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:15:00.360055 (+ 1775us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:15:00.360994 (+ 939us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:15:00.362902 (+ 1908us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:15:00.363392 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:15:00.363508 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:15:00.366954 (+ 3446us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:15:00.366970 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:15:00.366987 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:15:00.367033 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:15:00.371196 (+ 4163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:15:00.372191 (+ 995us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:15:00.372200 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:15:00.372202 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:15:00.372449 (+ 247us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:15:00.372966 (+ 517us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:15:00.372971 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:15:00.372973 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:15:00.373471 (+ 498us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:15:00.373765 (+ 294us) server_negotiation.cc:1035] Waiting for connection context
0928 02:15:00.374093 (+ 328us) server_negotiation.cc:299] Negotiation successful
0928 02:15:00.374318 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260928 02:15:00.375504 23483 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 02:15:00.492931 7085 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTlaUuc/build/asan/bin/kudu with pid 23360
I20260928 02:15:00.513737 7085 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskTlaUuc/build/asan/bin/kudu pid 23491
I20260928 02:15:00.530270 23632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:15:00.514672 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57552 (local address 127.6.235.67:37203)
0928 02:15:00.514882 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0928 02:15:00.514887 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 02:15:00.514907 (+ 20us) server_negotiation.cc:407] Connection header received
0928 02:15:00.515091 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:15:00.515095 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:15:00.515186 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:15:00.515281 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:15:00.516754 (+ 1473us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:15:00.517716 (+ 962us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:15:00.519472 (+ 1756us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:15:00.519898 (+ 426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:15:00.520007 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:15:00.523417 (+ 3410us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:15:00.523481 (+ 64us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:15:00.523501 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:15:00.523545 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:15:00.526898 (+ 3353us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:15:00.527823 (+ 925us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:15:00.527830 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:15:00.527832 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:15:00.528003 (+ 171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:15:00.528441 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:15:00.528447 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:15:00.528451 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:15:00.529034 (+ 583us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:15:00.529281 (+ 247us) server_negotiation.cc:1035] Waiting for connection context
0928 02:15:00.529638 (+ 357us) server_negotiation.cc:299] Negotiation successful
0928 02:15:00.529830 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260928 02:15:00.531116 23614 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 02:15:00.621531 7085 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTlaUuc/build/asan/bin/kudu with pid 23491
I20260928 02:15:00.639631 7085 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskTlaUuc/build/asan/bin/kudu pid 23138
I20260928 02:15:00.640887 23201 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 02:15:00.784662 7085 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTlaUuc/build/asan/bin/kudu with pid 23138
I20260928 02:15:20.810859 7085 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTlaUuc/build/asan/bin/kudu
/tmp/dist-test-taskTlaUuc/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790561432251112-7085-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790561432251112-7085-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790561432251112-7085-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790561432251112-7085-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.235.126:46725
--webserver_interface=127.6.235.126
--webserver_port=46735
--keytab_file=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790561432251112-7085-0/krb5kdc/kudu_127.6.235.126.keytab
--principal=kudu/127.6.235.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.235.84:46687
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.235.126:46725
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790561432251112-7085-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790561432251112-7085-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790561432251112-7085-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:15:21.169916 23665 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:15:21.170261 23665 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:15:21.170367 23665 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:15:21.177841 23665 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260928 02:15:21.177953 23665 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260928 02:15:21.177990 23665 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260928 02:15:21.178036 23665 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 02:15:21.178092 23665 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:15:21.178162 23665 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:15:21.178208 23665 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 02:15:21.178262 23665 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 02:15:21.188537 23665 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.235.84:46687
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790561432251112-7085-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790561432251112-7085-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.6.235.126:46725
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790561432251112-7085-0/krb5kdc/kudu_127.6.235.126.keytab
--principal=kudu/127.6.235.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.235.126:46725
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790561432251112-7085-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.235.126
--webserver_port=46735
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790561432251112-7085-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 00:51:59 UTC on ccycloud.quasar-axgkwn.root.comops.site
build id 4676464
ASAN enabled
I20260928 02:15:21.190006 23665 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:15:21.191366 23665 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:15:21.203882 23673 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:15:21.204003 23670 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:15:21.203929 23671 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:15:21.205281 23665 server_base.cc:1034] running on GCE node
I20260928 02:15:21.206291 23665 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:15:21.207985 23665 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:15:21.209288 23665 hybrid_clock.cc:648] HybridClock initialized: now 1790561721209229 us; error 56 us; skew 500 ppm
I20260928 02:15:21.209729 23665 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 02:15:21 dist-test-slave-86gv krb5kdc[22587](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561721, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.235.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:15:21.213385 23665 init.cc:377] Logged in from keytab as kudu/127.6.235.126@KRBTEST.COM (short username kudu)
I20260928 02:15:21.215116 23665 webserver.cc:460] Webserver started at http://127.6.235.126:46735/ using document root <none> and password file <none>
I20260928 02:15:21.215934 23665 fs_manager.cc:359] Metadata directory not provided
I20260928 02:15:21.216081 23665 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:15:21.222057 23665 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20260928 02:15:21.225167 23680 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:15:21.226627 23665 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260928 02:15:21.226843 23665 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790561432251112-7085-0/minicluster-data/master-0/data,/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790561432251112-7085-0/minicluster-data/master-0/wal
uuid: "2bc05ba1194e4a02b77ee3f8d3ef5771"
format_stamp: "Formatted at 2026-09-28 02:14:34 on dist-test-slave-86gv"
server_key: "fc8fbdce5fc05aef1c429652b161332b"
server_key_iv: "60e6e9c33bd1190bbbb6c468bc6f8a60"
server_key_version: "encryptionkey@0"
I20260928 02:15:21.227908 23665 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790561432251112-7085-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790561432251112-7085-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790561432251112-7085-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:15:21.243786 23665 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:15:21.248158 23665 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:15:21.248484 23665 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:15:21.268013 23665 rpc_server.cc:307] RPC server started. Bound to: 127.6.235.126:46725
I20260928 02:15:21.268046 23731 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.235.126:46725 every 8 connection(s)
I20260928 02:15:21.269789 23665 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790561432251112-7085-0/minicluster-data/master-0/data/info.pb
I20260928 02:15:21.274752 7085 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTlaUuc/build/asan/bin/kudu as pid 23665
I20260928 02:15:21.275790 7085 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTlaUuc/build/asan/bin/kudu
/tmp/dist-test-taskTlaUuc/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790561432251112-7085-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790561432251112-7085-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790561432251112-7085-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790561432251112-7085-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.235.65:33947
--local_ip_for_outbound_sockets=127.6.235.65
--tserver_master_addrs=127.6.235.126:46725
--webserver_port=43859
--webserver_interface=127.6.235.65
--keytab_file=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790561432251112-7085-0/krb5kdc/kudu_127.6.235.65.keytab
--principal=kudu/127.6.235.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.235.84:46687
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790561432251112-7085-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790561432251112-7085-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790561432251112-7085-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 02:15:21.278489 23732 sys_catalog.cc:263] Verifying existing consensus state
I20260928 02:15:21.284037 23732 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2bc05ba1194e4a02b77ee3f8d3ef5771: Bootstrap starting.
I20260928 02:15:21.306068 23732 log.cc:824] T 00000000000000000000000000000000 P 2bc05ba1194e4a02b77ee3f8d3ef5771: Log is configured to *not* fsync() on all Append() calls
I20260928 02:15:21.323136 23732 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2bc05ba1194e4a02b77ee3f8d3ef5771: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=8 ignored=0} mutations{seen=8 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 02:15:21.323747 23732 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2bc05ba1194e4a02b77ee3f8d3ef5771: Bootstrap complete.
I20260928 02:15:21.333838 23732 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2bc05ba1194e4a02b77ee3f8d3ef5771 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2bc05ba1194e4a02b77ee3f8d3ef5771" member_type: VOTER last_known_addr { host: "127.6.235.126" port: 46725 } }
I20260928 02:15:21.334880 23732 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2bc05ba1194e4a02b77ee3f8d3ef5771 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2bc05ba1194e4a02b77ee3f8d3ef5771, State: Initialized, Role: FOLLOWER
I20260928 02:15:21.335563 23732 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2bc05ba1194e4a02b77ee3f8d3ef5771 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2bc05ba1194e4a02b77ee3f8d3ef5771" member_type: VOTER last_known_addr { host: "127.6.235.126" port: 46725 } }
I20260928 02:15:21.335820 23732 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2bc05ba1194e4a02b77ee3f8d3ef5771 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 02:15:21.335969 23732 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2bc05ba1194e4a02b77ee3f8d3ef5771 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 02:15:21.336159 23732 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2bc05ba1194e4a02b77ee3f8d3ef5771 [term 2 FOLLOWER]: Advancing to term 3
I20260928 02:15:21.340552 23732 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2bc05ba1194e4a02b77ee3f8d3ef5771 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2bc05ba1194e4a02b77ee3f8d3ef5771" member_type: VOTER last_known_addr { host: "127.6.235.126" port: 46725 } }
I20260928 02:15:21.341144 23732 leader_election.cc:302] T 00000000000000000000000000000000 P 2bc05ba1194e4a02b77ee3f8d3ef5771 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2bc05ba1194e4a02b77ee3f8d3ef5771; no voters:
I20260928 02:15:21.341846 23732 leader_election.cc:288] T 00000000000000000000000000000000 P 2bc05ba1194e4a02b77ee3f8d3ef5771 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260928 02:15:21.341988 23736 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2bc05ba1194e4a02b77ee3f8d3ef5771 [term 3 FOLLOWER]: Leader election won for term 3
I20260928 02:15:21.343262 23736 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2bc05ba1194e4a02b77ee3f8d3ef5771 [term 3 LEADER]: Becoming Leader. State: Replica: 2bc05ba1194e4a02b77ee3f8d3ef5771, State: Running, Role: LEADER
I20260928 02:15:21.343925 23736 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2bc05ba1194e4a02b77ee3f8d3ef5771 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2bc05ba1194e4a02b77ee3f8d3ef5771" member_type: VOTER last_known_addr { host: "127.6.235.126" port: 46725 } }
I20260928 02:15:21.344215 23732 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2bc05ba1194e4a02b77ee3f8d3ef5771 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 02:15:21.346570 23738 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2bc05ba1194e4a02b77ee3f8d3ef5771 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2bc05ba1194e4a02b77ee3f8d3ef5771. Latest consensus state: current_term: 3 leader_uuid: "2bc05ba1194e4a02b77ee3f8d3ef5771" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2bc05ba1194e4a02b77ee3f8d3ef5771" member_type: VOTER last_known_addr { host: "127.6.235.126" port: 46725 } } }
I20260928 02:15:21.346931 23738 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2bc05ba1194e4a02b77ee3f8d3ef5771 [sys.catalog]: This master's current role is: LEADER
I20260928 02:15:21.347527 23737 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2bc05ba1194e4a02b77ee3f8d3ef5771 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "2bc05ba1194e4a02b77ee3f8d3ef5771" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2bc05ba1194e4a02b77ee3f8d3ef5771" member_type: VOTER last_known_addr { host: "127.6.235.126" port: 46725 } } }
I20260928 02:15:21.347810 23737 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2bc05ba1194e4a02b77ee3f8d3ef5771 [sys.catalog]: This master's current role is: LEADER
I20260928 02:15:21.350389 23743 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 02:15:21.359899 23743 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 02:15:21.361706 23743 catalog_manager.cc:1259] Loaded cluster ID: b4531193cd854a578528a141688183d9
I20260928 02:15:21.361913 23743 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 02:15:21.366421 23743 catalog_manager.cc:1504] Loading token signing keys...
I20260928 02:15:21.368973 23743 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 2bc05ba1194e4a02b77ee3f8d3ef5771: Loaded TSK: 0
I20260928 02:15:21.369235 23743 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 2bc05ba1194e4a02b77ee3f8d3ef5771: Loaded TSK: 1
F20260928 02:15:21.370687 23743 token_signing_key.cc:66] Check failed: _s.ok() Bad status: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f22dcb1b9ad google::LogMessage::Fail() at ??:0
@ 0x7f22dcb1e88a google::LogMessage::SendToLog() at ??:0
@ 0x7f22dcb1b4de google::LogMessage::Flush() at ??:0
@ 0x7f22dcb1ca19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f22e087f7d2 kudu::security::TokenSigningPrivateKey::TokenSigningPrivateKey() at ??:0
@ 0x7f22e086e73e kudu::security::TokenSigner::ImportKeys() at ??:0
@ 0x7f22f254e1c0 kudu::master::CatalogManager::LoadTskEntries() at ??:0
@ 0x7f22f254d783 kudu::master::CatalogManager::InitTokenSigner() at ??:0
@ 0x7f22f25d92a7 kudu::master::CatalogManager::PrepareForLeadershipTask()::$_10::operator()() at ??:0
@ 0x7f22f25d90c4 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x7f22f26235e9 std::function<>::operator()() at ??:0
@ 0x7f22f2551cdd kudu::master::CatalogManager::PrepareForLeadershipTask()::$_6::operator()() at ??:0
@ 0x7f22f254fc36 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f22f25d7610 kudu::master::CatalogManager::ElectedAsLeaderCb()::$_2::operator()() at ??:0
@ 0x7f22f25d7439 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7f22df767345 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f22df76a91c kudu::ThreadPool::CreateThread()::$_2::operator()() at ??:0
@ 0x7f22df76a799 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7f22df745eaf kudu::Thread::SuperviseThread() at ??:0
@ 0x7f22e2a1f6db start_thread at ??:0
@ 0x7f22db9b561f clone at ??:0
W20260928 02:15:21.677187 23734 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:15:21.677557 23734 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:15:21.677733 23734 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:15:21.685623 23734 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:15:21.685804 23734 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:15:21.685997 23734 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.235.65
I20260928 02:15:21.696487 23734 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.235.84:46687
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790561432251112-7085-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790561432251112-7085-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790561432251112-7085-0/krb5kdc/kudu_127.6.235.65.keytab
--principal=kudu/127.6.235.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.235.65:33947
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790561432251112-7085-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.6.235.65
--webserver_port=43859
--tserver_master_addrs=127.6.235.126:46725
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.235.65
--log_dir=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790561432251112-7085-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 00:51:59 UTC on ccycloud.quasar-axgkwn.root.comops.site
build id 4676464
ASAN enabled
I20260928 02:15:21.697912 23734 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:15:21.699312 23734 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:15:21.713601 23782 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:15:21.713621 23780 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:15:21.713683 23779 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:15:21.714745 23734 server_base.cc:1034] running on GCE node
I20260928 02:15:21.715436 23734 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:15:21.716445 23734 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:15:21.717691 23734 hybrid_clock.cc:648] HybridClock initialized: now 1790561721717647 us; error 37 us; skew 500 ppm
I20260928 02:15:21.718007 23734 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 02:15:21 dist-test-slave-86gv krb5kdc[22587](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561721, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.235.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:15:21.721383 23734 init.cc:377] Logged in from keytab as kudu/127.6.235.65@KRBTEST.COM (short username kudu)
I20260928 02:15:21.722954 23734 webserver.cc:460] Webserver started at http://127.6.235.65:43859/ using document root <none> and password file <none>
I20260928 02:15:21.723657 23734 fs_manager.cc:359] Metadata directory not provided
I20260928 02:15:21.723762 23734 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:15:21.729684 23734 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20260928 02:15:21.732576 23789 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:15:21.734350 23734 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260928 02:15:21.734596 23734 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790561432251112-7085-0/minicluster-data/ts-0/data,/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790561432251112-7085-0/minicluster-data/ts-0/wal
uuid: "c8d284c745a3487e80f359a662875c9d"
format_stamp: "Formatted at 2026-09-28 02:14:35 on dist-test-slave-86gv"
server_key: "585d0e117c6544ebf3316128d4952e70"
server_key_iv: "edf8eecd2aaa84f5fa0b8e26e372d834"
server_key_version: "encryptionkey@0"
I20260928 02:15:21.735831 23734 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790561432251112-7085-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790561432251112-7085-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790561432251112-7085-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:15:21.759110 23734 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:15:21.763973 23734 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:15:21.764292 23734 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:15:21.765460 23734 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:15:21.766981 23734 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:15:21.767071 23734 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:15:21.767236 23734 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:15:21.767331 23734 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:15:21.805536 23734 rpc_server.cc:307] RPC server started. Bound to: 127.6.235.65:33947
I20260928 02:15:21.805634 23901 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.235.65:33947 every 8 connection(s)
I20260928 02:15:21.807202 23734 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790561432251112-7085-0/minicluster-data/ts-0/data/info.pb
I20260928 02:15:21.812479 23904 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:15:21.810449 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.235.126:46725 (local address 127.6.235.65:53683)
0928 02:15:21.811329 (+ 880us) negotiation.cc:107] Waiting for socket to connect
0928 02:15:21.811563 (+ 234us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.235.126:46725: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":621,"thread_start_us":424,"threads_started":1}
W20260928 02:15:21.813357 23902 heartbeater.cc:644] Failed to heartbeat to 127.6.235.126:46725 (0 consecutive failures): Network error: Failed to ping master at 127.6.235.126:46725: Client connection negotiation failed: client connection to 127.6.235.126:46725: connect: Connection refused (error 111)
I20260928 02:15:21.814286 23904 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:15:21.813923 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.235.126:46725 (local address 127.6.235.65:38507)
0928 02:15:21.814090 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0928 02:15:21.814173 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.235.126:46725: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
I20260928 02:15:21.815196 23904 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:15:21.814826 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.235.126:46725 (local address 127.6.235.65:44481)
0928 02:15:21.814997 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0928 02:15:21.815080 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.235.126:46725: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260928 02:15:21.815433 23902 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260928 02:15:21.816638 7085 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTlaUuc/build/asan/bin/kudu as pid 23734
I20260928 02:15:21.817354 7085 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTlaUuc/build/asan/bin/kudu
/tmp/dist-test-taskTlaUuc/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790561432251112-7085-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790561432251112-7085-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790561432251112-7085-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790561432251112-7085-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.235.66:46099
--local_ip_for_outbound_sockets=127.6.235.66
--tserver_master_addrs=127.6.235.126:46725
--webserver_port=43345
--webserver_interface=127.6.235.66
--keytab_file=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790561432251112-7085-0/krb5kdc/kudu_127.6.235.66.keytab
--principal=kudu/127.6.235.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.235.84:46687
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790561432251112-7085-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790561432251112-7085-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790561432251112-7085-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:15:22.173550 23905 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:15:22.173909 23905 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:15:22.174082 23905 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:15:22.182121 23905 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:15:22.182250 23905 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:15:22.182439 23905 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.235.66
I20260928 02:15:22.192946 23905 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.235.84:46687
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790561432251112-7085-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790561432251112-7085-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790561432251112-7085-0/krb5kdc/kudu_127.6.235.66.keytab
--principal=kudu/127.6.235.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.235.66:46099
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790561432251112-7085-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.6.235.66
--webserver_port=43345
--tserver_master_addrs=127.6.235.126:46725
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.235.66
--log_dir=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790561432251112-7085-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 00:51:59 UTC on ccycloud.quasar-axgkwn.root.comops.site
build id 4676464
ASAN enabled
I20260928 02:15:22.194438 23905 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:15:22.195852 23905 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:15:22.210717 23911 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:15:22.210747 23913 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:15:22.210706 23910 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:15:22.211899 23905 server_base.cc:1034] running on GCE node
I20260928 02:15:22.212581 23905 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:15:22.213719 23905 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:15:22.214926 23905 hybrid_clock.cc:648] HybridClock initialized: now 1790561722214876 us; error 40 us; skew 500 ppm
I20260928 02:15:22.215265 23905 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 02:15:22 dist-test-slave-86gv krb5kdc[22587](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561722, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.235.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:15:22.218642 23905 init.cc:377] Logged in from keytab as kudu/127.6.235.66@KRBTEST.COM (short username kudu)
I20260928 02:15:22.220311 23905 webserver.cc:460] Webserver started at http://127.6.235.66:43345/ using document root <none> and password file <none>
I20260928 02:15:22.221129 23905 fs_manager.cc:359] Metadata directory not provided
I20260928 02:15:22.221323 23905 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:15:22.227140 23905 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.002s
I20260928 02:15:22.230520 23920 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:15:22.232424 23905 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260928 02:15:22.232703 23905 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790561432251112-7085-0/minicluster-data/ts-1/data,/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790561432251112-7085-0/minicluster-data/ts-1/wal
uuid: "6c165d86953345f6a45e9812e72c1b70"
format_stamp: "Formatted at 2026-09-28 02:14:36 on dist-test-slave-86gv"
server_key: "faed87a17ceb68956d872503da8c75b5"
server_key_iv: "ad63fd5a32bacd6faf8da18b56601cbe"
server_key_version: "encryptionkey@0"
I20260928 02:15:22.233908 23905 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790561432251112-7085-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790561432251112-7085-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790561432251112-7085-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:15:22.255436 23905 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:15:22.260061 23905 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:15:22.260377 23905 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:15:22.261466 23905 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:15:22.262900 23905 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:15:22.262989 23905 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:15:22.263168 23905 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:15:22.263269 23905 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:15:22.297715 23905 rpc_server.cc:307] RPC server started. Bound to: 127.6.235.66:46099
I20260928 02:15:22.297767 24032 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.235.66:46099 every 8 connection(s)
I20260928 02:15:22.299252 23905 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790561432251112-7085-0/minicluster-data/ts-1/data/info.pb
I20260928 02:15:22.304432 7085 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTlaUuc/build/asan/bin/kudu as pid 23905
I20260928 02:15:22.304863 24035 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:15:22.302636 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.235.126:46725 (local address 127.6.235.66:39769)
0928 02:15:22.303403 (+ 767us) negotiation.cc:107] Waiting for socket to connect
0928 02:15:22.303630 (+ 227us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.235.126:46725: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":478,"thread_start_us":286,"threads_started":1}
I20260928 02:15:22.305320 7085 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTlaUuc/build/asan/bin/kudu
/tmp/dist-test-taskTlaUuc/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790561432251112-7085-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790561432251112-7085-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790561432251112-7085-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790561432251112-7085-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.235.67:37203
--local_ip_for_outbound_sockets=127.6.235.67
--tserver_master_addrs=127.6.235.126:46725
--webserver_port=34857
--webserver_interface=127.6.235.67
--keytab_file=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790561432251112-7085-0/krb5kdc/kudu_127.6.235.67.keytab
--principal=kudu/127.6.235.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.235.84:46687
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790561432251112-7085-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790561432251112-7085-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790561432251112-7085-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:15:22.305560 24033 heartbeater.cc:644] Failed to heartbeat to 127.6.235.126:46725 (0 consecutive failures): Network error: Failed to ping master at 127.6.235.126:46725: Client connection negotiation failed: client connection to 127.6.235.126:46725: connect: Connection refused (error 111)
I20260928 02:15:22.306543 24035 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:15:22.306076 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.235.126:46725 (local address 127.6.235.66:40287)
0928 02:15:22.306275 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0928 02:15:22.306386 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.235.126:46725: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
I20260928 02:15:22.307526 24035 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:15:22.307081 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.235.126:46725 (local address 127.6.235.66:45191)
0928 02:15:22.307261 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0928 02:15:22.307370 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.235.126:46725: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
W20260928 02:15:22.307895 24033 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260928 02:15:22.649575 24036 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:15:22.650094 24036 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:15:22.650270 24036 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:15:22.659209 24036 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:15:22.659399 24036 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:15:22.659569 24036 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.235.67
I20260928 02:15:22.669922 24036 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.235.84:46687
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790561432251112-7085-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790561432251112-7085-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790561432251112-7085-0/krb5kdc/kudu_127.6.235.67.keytab
--principal=kudu/127.6.235.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.235.67:37203
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790561432251112-7085-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.6.235.67
--webserver_port=34857
--tserver_master_addrs=127.6.235.126:46725
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.235.67
--log_dir=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790561432251112-7085-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 00:51:59 UTC on ccycloud.quasar-axgkwn.root.comops.site
build id 4676464
ASAN enabled
I20260928 02:15:22.671280 24036 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:15:22.672610 24036 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:15:22.687647 24041 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:15:22.687593 24044 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:15:22.687668 24042 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:15:22.688942 24036 server_base.cc:1034] running on GCE node
I20260928 02:15:22.689668 24036 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:15:22.690773 24036 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:15:22.692051 24036 hybrid_clock.cc:648] HybridClock initialized: now 1790561722691983 us; error 59 us; skew 500 ppm
I20260928 02:15:22.692481 24036 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 02:15:22 dist-test-slave-86gv krb5kdc[22587](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561722, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.235.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:15:22.696022 24036 init.cc:377] Logged in from keytab as kudu/127.6.235.67@KRBTEST.COM (short username kudu)
I20260928 02:15:22.697781 24036 webserver.cc:460] Webserver started at http://127.6.235.67:34857/ using document root <none> and password file <none>
I20260928 02:15:22.698563 24036 fs_manager.cc:359] Metadata directory not provided
I20260928 02:15:22.698720 24036 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:15:22.704483 24036 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20260928 02:15:22.707470 24051 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:15:22.709000 24036 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 02:15:22.709192 24036 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790561432251112-7085-0/minicluster-data/ts-2/data,/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790561432251112-7085-0/minicluster-data/ts-2/wal
uuid: "613712f91b1348c29b808745f72d275a"
format_stamp: "Formatted at 2026-09-28 02:14:36 on dist-test-slave-86gv"
server_key: "0f80c683ff15b55d4fcfe676df3711df"
server_key_iv: "68eaf86367030440b44df4090297600d"
server_key_version: "encryptionkey@0"
I20260928 02:15:22.710187 24036 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790561432251112-7085-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790561432251112-7085-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790561432251112-7085-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:15:22.723754 24036 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:15:22.728312 24036 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:15:22.728624 24036 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:15:22.729844 24036 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:15:22.731379 24036 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:15:22.731513 24036 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:15:22.731645 24036 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:15:22.731731 24036 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:15:22.776752 24036 rpc_server.cc:307] RPC server started. Bound to: 127.6.235.67:37203
I20260928 02:15:22.776851 24163 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.235.67:37203 every 8 connection(s)
I20260928 02:15:22.778303 24036 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790561432251112-7085-0/minicluster-data/ts-2/data/info.pb
I20260928 02:15:22.782475 7085 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTlaUuc/build/asan/bin/kudu as pid 24036
I20260928 02:15:22.783834 24166 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:15:22.781557 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.235.126:46725 (local address 127.6.235.67:52663)
0928 02:15:22.782358 (+ 801us) negotiation.cc:107] Waiting for socket to connect
0928 02:15:22.782623 (+ 265us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.235.126:46725: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":463,"thread_start_us":282,"threads_started":1}
W20260928 02:15:22.784602 24164 heartbeater.cc:644] Failed to heartbeat to 127.6.235.126:46725 (0 consecutive failures): Network error: Failed to ping master at 127.6.235.126:46725: Client connection negotiation failed: client connection to 127.6.235.126:46725: connect: Connection refused (error 111)
I20260928 02:15:22.785607 24166 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:15:22.785141 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.235.126:46725 (local address 127.6.235.67:54109)
0928 02:15:22.785402 (+ 261us) negotiation.cc:107] Waiting for socket to connect
0928 02:15:22.785481 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.235.126:46725: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":160}
I20260928 02:15:22.786646 24166 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:15:22.786207 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.235.126:46725 (local address 127.6.235.67:56533)
0928 02:15:22.786443 (+ 236us) negotiation.cc:107] Waiting for socket to connect
0928 02:15:22.786524 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.235.126:46725: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":140}
W20260928 02:15:22.786932 24164 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260928 02:15:22.817062 24168 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:15:22.816069 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.235.126:46725 (local address 127.6.235.65:54009)
0928 02:15:22.816595 (+ 526us) negotiation.cc:107] Waiting for socket to connect
0928 02:15:22.816752 (+ 157us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.235.126:46725: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":403,"thread_start_us":268,"threads_started":1}
I20260928 02:15:23.309691 24169 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:15:23.308627 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.235.126:46725 (local address 127.6.235.66:43567)
0928 02:15:23.309208 (+ 581us) negotiation.cc:107] Waiting for socket to connect
0928 02:15:23.309413 (+ 205us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.235.126:46725: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":453,"thread_start_us":329,"threads_started":1}
I20260928 02:15:23.788635 24170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:15:23.787703 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.235.126:46725 (local address 127.6.235.67:46607)
0928 02:15:23.788238 (+ 535us) negotiation.cc:107] Waiting for socket to connect
0928 02:15:23.788378 (+ 140us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.235.126:46725: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":418,"thread_start_us":276,"threads_started":1}
I20260928 02:15:23.819046 24171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:15:23.818238 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.235.126:46725 (local address 127.6.235.65:40041)
0928 02:15:23.818781 (+ 543us) negotiation.cc:107] Waiting for socket to connect
0928 02:15:23.818874 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.235.126:46725: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":419,"thread_start_us":260,"threads_started":1}
I20260928 02:15:24.311532 24172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:15:24.310736 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.235.126:46725 (local address 127.6.235.66:47353)
0928 02:15:24.311273 (+ 537us) negotiation.cc:107] Waiting for socket to connect
0928 02:15:24.311354 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.235.126:46725: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":436,"thread_start_us":277,"threads_started":1}
I20260928 02:15:24.790498 24173 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:15:24.789710 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.235.126:46725 (local address 127.6.235.67:46097)
0928 02:15:24.790224 (+ 514us) negotiation.cc:107] Waiting for socket to connect
0928 02:15:24.790320 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.235.126:46725: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":389,"thread_start_us":256,"threads_started":1}
I20260928 02:15:24.821094 24174 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:15:24.820251 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.235.126:46725 (local address 127.6.235.65:55363)
0928 02:15:24.820778 (+ 527us) negotiation.cc:107] Waiting for socket to connect
0928 02:15:24.820878 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.235.126:46725: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":383,"thread_start_us":268,"threads_started":1}
I20260928 02:15:25.313591 24175 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:15:25.312654 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.235.126:46725 (local address 127.6.235.66:33539)
0928 02:15:25.313298 (+ 644us) negotiation.cc:107] Waiting for socket to connect
0928 02:15:25.313401 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.235.126:46725: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":449,"thread_start_us":327,"threads_started":1}
I20260928 02:15:25.792322 24176 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:15:25.791461 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.235.126:46725 (local address 127.6.235.67:33017)
0928 02:15:25.792013 (+ 552us) negotiation.cc:107] Waiting for socket to connect
0928 02:15:25.792105 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.235.126:46725: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":422,"thread_start_us":291,"threads_started":1}
I20260928 02:15:25.822845 24177 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:15:25.822094 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.235.126:46725 (local address 127.6.235.65:44515)
0928 02:15:25.822586 (+ 492us) negotiation.cc:107] Waiting for socket to connect
0928 02:15:25.822673 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.235.126:46725: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":379,"thread_start_us":261,"threads_started":1}
I20260928 02:15:26.315441 24178 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:15:26.314600 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.235.126:46725 (local address 127.6.235.66:46235)
0928 02:15:26.315113 (+ 513us) negotiation.cc:107] Waiting for socket to connect
0928 02:15:26.315199 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.235.126:46725: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":408,"thread_start_us":282,"threads_started":1}
I20260928 02:15:26.794296 24179 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:15:26.793494 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.235.126:46725 (local address 127.6.235.67:41607)
0928 02:15:26.794010 (+ 516us) negotiation.cc:107] Waiting for socket to connect
0928 02:15:26.794100 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.235.126:46725: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":406,"thread_start_us":257,"threads_started":1}
I20260928 02:15:26.824882 24180 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:15:26.823983 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.235.126:46725 (local address 127.6.235.65:39871)
0928 02:15:26.824545 (+ 562us) negotiation.cc:107] Waiting for socket to connect
0928 02:15:26.824632 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.235.126:46725: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":457,"thread_start_us":279,"threads_started":1}
I20260928 02:15:27.317265 24181 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:15:27.316422 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.235.126:46725 (local address 127.6.235.66:55443)
0928 02:15:27.316949 (+ 527us) negotiation.cc:107] Waiting for socket to connect
0928 02:15:27.317057 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.235.126:46725: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":366,"thread_start_us":228,"threads_started":1}
I20260928 02:15:27.796169 24182 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:15:27.795393 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.235.126:46725 (local address 127.6.235.67:47627)
0928 02:15:27.795912 (+ 519us) negotiation.cc:107] Waiting for socket to connect
0928 02:15:27.795995 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.235.126:46725: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":418,"thread_start_us":249,"threads_started":1}
I20260928 02:15:27.827059 24183 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:15:27.826148 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.235.126:46725 (local address 127.6.235.65:41759)
0928 02:15:27.826717 (+ 569us) negotiation.cc:107] Waiting for socket to connect
0928 02:15:27.826867 (+ 150us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.235.126:46725: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":436,"thread_start_us":302,"threads_started":1}
I20260928 02:15:28.319334 24184 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:15:28.318319 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.235.126:46725 (local address 127.6.235.66:45799)
0928 02:15:28.318933 (+ 614us) negotiation.cc:107] Waiting for socket to connect
0928 02:15:28.319031 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.235.126:46725: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":496,"thread_start_us":388,"threads_started":1}
I20260928 02:15:28.798157 24185 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:15:28.797393 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.235.126:46725 (local address 127.6.235.67:40321)
0928 02:15:28.797875 (+ 482us) negotiation.cc:107] Waiting for socket to connect
0928 02:15:28.797976 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.235.126:46725: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":357,"thread_start_us":227,"threads_started":1}
I20260928 02:15:28.829346 24186 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:15:28.828281 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.235.126:46725 (local address 127.6.235.65:41613)
0928 02:15:28.828876 (+ 595us) negotiation.cc:107] Waiting for socket to connect
0928 02:15:28.829060 (+ 184us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.235.126:46725: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":441,"thread_start_us":257,"threads_started":1}
I20260928 02:15:29.321633 24187 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:15:29.320503 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.235.126:46725 (local address 127.6.235.66:48129)
0928 02:15:29.321247 (+ 744us) negotiation.cc:107] Waiting for socket to connect
0928 02:15:29.321347 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.235.126:46725: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":520,"thread_start_us":257,"threads_started":1}
I20260928 02:15:29.800336 24188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:15:29.799300 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.235.126:46725 (local address 127.6.235.67:59805)
0928 02:15:29.799871 (+ 571us) negotiation.cc:107] Waiting for socket to connect
0928 02:15:29.799972 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.235.126:46725: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":446,"thread_start_us":249,"threads_started":1}
I20260928 02:15:29.831458 24189 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:15:29.830620 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.235.126:46725 (local address 127.6.235.65:44279)
0928 02:15:29.831180 (+ 560us) negotiation.cc:107] Waiting for socket to connect
0928 02:15:29.831284 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.235.126:46725: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":422,"thread_start_us":273,"threads_started":1}
I20260928 02:15:30.323829 24190 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:15:30.322936 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.235.126:46725 (local address 127.6.235.66:54387)
0928 02:15:30.323517 (+ 581us) negotiation.cc:107] Waiting for socket to connect
0928 02:15:30.323626 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.235.126:46725: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":468,"thread_start_us":342,"threads_started":1}
I20260928 02:15:30.802454 24191 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:15:30.801631 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.235.126:46725 (local address 127.6.235.67:55239)
0928 02:15:30.802185 (+ 554us) negotiation.cc:107] Waiting for socket to connect
0928 02:15:30.802280 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.235.126:46725: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":439,"thread_start_us":310,"threads_started":1}
I20260928 02:15:30.833426 24192 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:15:30.832419 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.235.126:46725 (local address 127.6.235.65:56827)
0928 02:15:30.833058 (+ 639us) negotiation.cc:107] Waiting for socket to connect
0928 02:15:30.833229 (+ 171us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.235.126:46725: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":438,"thread_start_us":293,"threads_started":1}
I20260928 02:15:31.326061 24193 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:15:31.325203 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.235.126:46725 (local address 127.6.235.66:53749)
0928 02:15:31.325762 (+ 559us) negotiation.cc:107] Waiting for socket to connect
0928 02:15:31.325863 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.235.126:46725: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":440,"thread_start_us":272,"threads_started":1}
I20260928 02:15:31.804427 24194 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:15:31.803577 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.235.126:46725 (local address 127.6.235.67:36623)
0928 02:15:31.804132 (+ 555us) negotiation.cc:107] Waiting for socket to connect
0928 02:15:31.804236 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.235.126:46725: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":444,"thread_start_us":288,"threads_started":1}
I20260928 02:15:31.835649 24195 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:15:31.834624 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.235.126:46725 (local address 127.6.235.65:47251)
0928 02:15:31.835295 (+ 671us) negotiation.cc:107] Waiting for socket to connect
0928 02:15:31.835425 (+ 130us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.235.126:46725: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":486,"thread_start_us":352,"threads_started":1}
I20260928 02:15:32.328061 24196 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:15:32.327298 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.235.126:46725 (local address 127.6.235.66:40565)
0928 02:15:32.327799 (+ 501us) negotiation.cc:107] Waiting for socket to connect
0928 02:15:32.327892 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.235.126:46725: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":380,"thread_start_us":260,"threads_started":1}
I20260928 02:15:32.806659 24197 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:15:32.805686 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.235.126:46725 (local address 127.6.235.67:41325)
0928 02:15:32.806343 (+ 657us) negotiation.cc:107] Waiting for socket to connect
0928 02:15:32.806465 (+ 122us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.235.126:46725: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":516,"thread_start_us":251,"threads_started":1}
I20260928 02:15:32.837591 24198 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:15:32.836690 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.235.126:46725 (local address 127.6.235.65:34229)
0928 02:15:32.837287 (+ 597us) negotiation.cc:107] Waiting for socket to connect
0928 02:15:32.837395 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.235.126:46725: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":480,"thread_start_us":287,"threads_started":1}
I20260928 02:15:33.330102 24199 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:15:33.329310 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.235.126:46725 (local address 127.6.235.66:47357)
0928 02:15:33.329802 (+ 492us) negotiation.cc:107] Waiting for socket to connect
0928 02:15:33.329907 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.235.126:46725: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":379,"thread_start_us":236,"threads_started":1}
I20260928 02:15:33.808768 24200 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:15:33.807815 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.235.126:46725 (local address 127.6.235.67:38507)
0928 02:15:33.808375 (+ 560us) negotiation.cc:107] Waiting for socket to connect
0928 02:15:33.808523 (+ 148us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.235.126:46725: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":436,"thread_start_us":311,"threads_started":1}
I20260928 02:15:33.840086 24201 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:15:33.838966 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.235.126:46725 (local address 127.6.235.65:35203)
0928 02:15:33.839729 (+ 763us) negotiation.cc:107] Waiting for socket to connect
0928 02:15:33.839855 (+ 126us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.235.126:46725: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":608,"thread_start_us":495,"threads_started":1}
I20260928 02:15:34.332098 24202 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:15:34.331354 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.235.126:46725 (local address 127.6.235.66:57013)
0928 02:15:34.331838 (+ 484us) negotiation.cc:107] Waiting for socket to connect
0928 02:15:34.331929 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.235.126:46725: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":338,"thread_start_us":224,"threads_started":1}
I20260928 02:15:34.810546 24203 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:15:34.809847 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.235.126:46725 (local address 127.6.235.67:59823)
0928 02:15:34.810300 (+ 453us) negotiation.cc:107] Waiting for socket to connect
0928 02:15:34.810387 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.235.126:46725: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":332,"thread_start_us":218,"threads_started":1}
I20260928 02:15:34.842190 24204 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:15:34.841098 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.235.126:46725 (local address 127.6.235.65:39761)
0928 02:15:34.841797 (+ 699us) negotiation.cc:107] Waiting for socket to connect
0928 02:15:34.841908 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.235.126:46725: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":593,"thread_start_us":317,"threads_started":1}
I20260928 02:15:35.334053 24205 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:15:35.333231 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.235.126:46725 (local address 127.6.235.66:38861)
0928 02:15:35.333762 (+ 531us) negotiation.cc:107] Waiting for socket to connect
0928 02:15:35.333864 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.235.126:46725: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":423,"thread_start_us":251,"threads_started":1}
I20260928 02:15:35.812367 24206 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:15:35.811541 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.235.126:46725 (local address 127.6.235.67:53495)
0928 02:15:35.812061 (+ 520us) negotiation.cc:107] Waiting for socket to connect
0928 02:15:35.812169 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.235.126:46725: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":404,"thread_start_us":253,"threads_started":1}
I20260928 02:15:35.844074 24207 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:15:35.843293 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.235.126:46725 (local address 127.6.235.65:33845)
0928 02:15:35.843770 (+ 477us) negotiation.cc:107] Waiting for socket to connect
0928 02:15:35.843867 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.235.126:46725: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":359,"thread_start_us":245,"threads_started":1}
I20260928 02:15:36.335978 24208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:15:36.335206 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.235.126:46725 (local address 127.6.235.66:56489)
0928 02:15:36.335698 (+ 492us) negotiation.cc:107] Waiting for socket to connect
0928 02:15:36.335789 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.235.126:46725: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":373,"thread_start_us":255,"threads_started":1}
I20260928 02:15:36.814677 24209 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:15:36.813563 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.235.126:46725 (local address 127.6.235.67:36491)
0928 02:15:36.814319 (+ 756us) negotiation.cc:107] Waiting for socket to connect
0928 02:15:36.814417 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.235.126:46725: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":582,"thread_start_us":353,"threads_started":1}
I20260928 02:15:36.846215 24210 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:15:36.845393 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.235.126:46725 (local address 127.6.235.65:51663)
0928 02:15:36.845894 (+ 501us) negotiation.cc:107] Waiting for socket to connect
0928 02:15:36.845992 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.235.126:46725: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":387,"thread_start_us":277,"threads_started":1}
I20260928 02:15:37.338814 24211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:15:37.337328 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.235.126:46725 (local address 127.6.235.66:56229)
0928 02:15:37.338095 (+ 767us) negotiation.cc:107] Waiting for socket to connect
0928 02:15:37.338496 (+ 401us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.235.126:46725: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":565,"thread_start_us":439,"threads_started":1}
I20260928 02:15:37.793284 7085 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskTlaUuc/build/asan/bin/kudu pid 23734
I20260928 02:15:37.816756 24213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:15:37.815875 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.235.126:46725 (local address 127.6.235.67:60215)
0928 02:15:37.816476 (+ 601us) negotiation.cc:107] Waiting for socket to connect
0928 02:15:37.816567 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.235.126:46725: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":470,"thread_start_us":359,"threads_started":1}
I20260928 02:15:37.820547 24212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:15:37.794582 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51830 (local address 127.6.235.65:33947)
0928 02:15:37.795324 (+ 742us) server_negotiation.cc:206] Beginning negotiation
0928 02:15:37.795340 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0928 02:15:37.795383 (+ 43us) server_negotiation.cc:407] Connection header received
0928 02:15:37.797301 (+ 1918us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:15:37.797314 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:15:37.797805 (+ 491us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:15:37.798149 (+ 344us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:15:37.799770 (+ 1621us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:15:37.801336 (+ 1566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:15:37.802774 (+ 1438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:15:37.803504 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:15:37.803666 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:15:37.812326 (+ 8660us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:15:37.812337 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:15:37.812357 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:15:37.812410 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:15:37.816228 (+ 3818us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:15:37.817302 (+ 1074us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:15:37.817311 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:15:37.817326 (+ 15us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:15:37.817544 (+ 218us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:15:37.818047 (+ 503us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:15:37.818060 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:15:37.818064 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:15:37.818758 (+ 694us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:15:37.819138 (+ 380us) server_negotiation.cc:1035] Waiting for connection context
0928 02:15:37.819708 (+ 570us) server_negotiation.cc:299] Negotiation successful
0928 02:15:37.820223 (+ 515us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":532,"thread_start_us":358,"threads_started":1}
I20260928 02:15:37.822234 23897 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 02:15:37.913511 7085 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTlaUuc/build/asan/bin/kudu with pid 23734
I20260928 02:15:37.932339 7085 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskTlaUuc/build/asan/bin/kudu pid 23905
I20260928 02:15:37.951660 24216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:15:37.933556 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45462 (local address 127.6.235.66:46099)
0928 02:15:37.934084 (+ 528us) server_negotiation.cc:206] Beginning negotiation
0928 02:15:37.934099 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0928 02:15:37.934138 (+ 39us) server_negotiation.cc:407] Connection header received
0928 02:15:37.935745 (+ 1607us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:15:37.935755 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:15:37.936243 (+ 488us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:15:37.936660 (+ 417us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:15:37.938402 (+ 1742us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:15:37.939817 (+ 1415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:15:37.941194 (+ 1377us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:15:37.941927 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:15:37.942159 (+ 232us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:15:37.945031 (+ 2872us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:15:37.945042 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:15:37.945059 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:15:37.945107 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:15:37.948029 (+ 2922us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:15:37.948806 (+ 777us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:15:37.948813 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:15:37.948820 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:15:37.949072 (+ 252us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:15:37.949470 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:15:37.949476 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:15:37.949479 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:15:37.950076 (+ 597us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:15:37.950413 (+ 337us) server_negotiation.cc:1035] Waiting for connection context
0928 02:15:37.950920 (+ 507us) server_negotiation.cc:299] Negotiation successful
0928 02:15:37.951361 (+ 441us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":367,"thread_start_us":253,"threads_started":1}
I20260928 02:15:37.953485 24028 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 02:15:38.041816 7085 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTlaUuc/build/asan/bin/kudu with pid 23905
I20260928 02:15:38.060120 7085 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskTlaUuc/build/asan/bin/kudu pid 24036
I20260928 02:15:38.079761 24218 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:15:38.061329 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45692 (local address 127.6.235.67:37203)
0928 02:15:38.061961 (+ 632us) server_negotiation.cc:206] Beginning negotiation
0928 02:15:38.061975 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0928 02:15:38.062019 (+ 44us) server_negotiation.cc:407] Connection header received
0928 02:15:38.063753 (+ 1734us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:15:38.063764 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:15:38.064219 (+ 455us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:15:38.064523 (+ 304us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:15:38.066157 (+ 1634us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:15:38.067422 (+ 1265us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:15:38.068885 (+ 1463us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:15:38.069760 (+ 875us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:15:38.069926 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:15:38.072787 (+ 2861us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:15:38.072796 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:15:38.072815 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:15:38.072870 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:15:38.076297 (+ 3427us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:15:38.077198 (+ 901us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:15:38.077206 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:15:38.077214 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:15:38.077391 (+ 177us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:15:38.077744 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:15:38.077747 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:15:38.077749 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:15:38.078393 (+ 644us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:15:38.078642 (+ 249us) server_negotiation.cc:1035] Waiting for connection context
0928 02:15:38.079020 (+ 378us) server_negotiation.cc:299] Negotiation successful
0928 02:15:38.079447 (+ 427us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":471,"thread_start_us":321,"threads_started":1}
I20260928 02:15:38.081276 24159 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 02:15:38.168457 7085 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTlaUuc/build/asan/bin/kudu with pid 24036
2026-09-28T02:15:38Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedTSK (64981 ms)
[----------] 30 tests from SecurityITest (305874 ms total)
[----------] 6 tests from EncryptionPolicyTest
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/0
Loading random data
Initializing database '/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790561432251112-7085-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 02:15:38 dist-test-slave-86gv krb5kdc[24222](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 02:15:38 dist-test-slave-86gv krb5kdc[24222](info): set up 2 sockets
Sep 28 02:15:38 dist-test-slave-86gv krb5kdc[24222](info): commencing operation
krb5kdc: starting...
W20260928 02:15:39.313651 7085 mini_kdc.cc:121] Time spent starting KDC: real 1.103s user 0.001s sys 0.029s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790561432251112-7085-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790561432251112-7085-0/krb5kdc/test-user.keytab.
Sep 28 02:15:39 dist-test-slave-86gv krb5kdc[24222](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561739, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T02:15:39Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T02:15:39Z Disabled control of system clock
WARNING: no policy specified for kudu/127.6.235.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.235.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.235.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790561432251112-7085-0/krb5kdc/kudu_127.6.235.126.keytab.
Entry for principal kudu/127.6.235.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790561432251112-7085-0/krb5kdc/kudu_127.6.235.126.keytab.
I20260928 02:15:39.585664 7085 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTlaUuc/build/asan/bin/kudu
/tmp/dist-test-taskTlaUuc/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790561432251112-7085-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790561432251112-7085-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790561432251112-7085-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790561432251112-7085-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.235.126:46297
--webserver_interface=127.6.235.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790561432251112-7085-0/krb5kdc/kudu_127.6.235.126.keytab
--principal=kudu/127.6.235.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.235.84:39007
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.235.126:46297
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790561432251112-7085-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790561432251112-7085-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790561432251112-7085-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:15:39.934710 24235 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:15:39.935112 24235 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:15:39.935294 24235 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:15:39.942787 24235 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 02:15:39.942956 24235 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:15:39.943078 24235 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:15:39.943185 24235 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 02:15:39.943257 24235 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 02:15:39.953495 24235 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.235.84:39007
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790561432251112-7085-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790561432251112-7085-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.235.126:46297
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790561432251112-7085-0/krb5kdc/kudu_127.6.235.126.keytab
--principal=kudu/127.6.235.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.235.126:46297
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790561432251112-7085-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.235.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790561432251112-7085-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 00:51:59 UTC on ccycloud.quasar-axgkwn.root.comops.site
build id 4676464
ASAN enabled
I20260928 02:15:39.955142 24235 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:15:39.956559 24235 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:15:39.969396 24240 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:15:39.969425 24241 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:15:39.969513 24235 server_base.cc:1034] running on GCE node
W20260928 02:15:39.969417 24243 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:15:39.970882 24235 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:15:39.972380 24235 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:15:39.973641 24235 hybrid_clock.cc:648] HybridClock initialized: now 1790561739973652 us; error 128 us; skew 500 ppm
I20260928 02:15:39.974025 24235 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 02:15:39 dist-test-slave-86gv krb5kdc[24222](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561739, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.235.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:15:39.977325 24235 init.cc:377] Logged in from keytab as kudu/127.6.235.126@KRBTEST.COM (short username kudu)
I20260928 02:15:39.978861 24235 webserver.cc:460] Webserver started at http://127.6.235.126:45325/ using document root <none> and password file <none>
I20260928 02:15:39.979542 24235 fs_manager.cc:359] Metadata directory not provided
I20260928 02:15:39.979646 24235 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:15:39.979907 24235 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:15:39.983156 24235 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790561432251112-7085-0/minicluster-data/master-0/data/instance:
uuid: "b1442e6363434cd2ae8998baac5feb46"
format_stamp: "Formatted at 2026-09-28 02:15:39 on dist-test-slave-86gv"
server_key: "c6677f11c5786de3165b729e4d3bc1d1"
server_key_iv: "596fa3c1c4dc6ee6d7b67a7b74363b1a"
server_key_version: "encryptionkey@0"
I20260928 02:15:39.983912 24235 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790561432251112-7085-0/minicluster-data/master-0/wal/instance:
uuid: "b1442e6363434cd2ae8998baac5feb46"
format_stamp: "Formatted at 2026-09-28 02:15:39 on dist-test-slave-86gv"
server_key: "c6677f11c5786de3165b729e4d3bc1d1"
server_key_iv: "596fa3c1c4dc6ee6d7b67a7b74363b1a"
server_key_version: "encryptionkey@0"
I20260928 02:15:39.989749 24235 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260928 02:15:39.993660 24250 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:15:39.995329 24235 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260928 02:15:39.995535 24235 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790561432251112-7085-0/minicluster-data/master-0/data,/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790561432251112-7085-0/minicluster-data/master-0/wal
uuid: "b1442e6363434cd2ae8998baac5feb46"
format_stamp: "Formatted at 2026-09-28 02:15:39 on dist-test-slave-86gv"
server_key: "c6677f11c5786de3165b729e4d3bc1d1"
server_key_iv: "596fa3c1c4dc6ee6d7b67a7b74363b1a"
server_key_version: "encryptionkey@0"
I20260928 02:15:39.995775 24235 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790561432251112-7085-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790561432251112-7085-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790561432251112-7085-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:15:40.017069 24235 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:15:40.021946 24235 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:15:40.022272 24235 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:15:40.042233 24235 rpc_server.cc:307] RPC server started. Bound to: 127.6.235.126:46297
I20260928 02:15:40.042277 24301 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.235.126:46297 every 8 connection(s)
I20260928 02:15:40.043747 24235 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790561432251112-7085-0/minicluster-data/master-0/data/info.pb
I20260928 02:15:40.047827 24302 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:15:40.052659 7085 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTlaUuc/build/asan/bin/kudu as pid 24235
I20260928 02:15:40.052925 7085 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790561432251112-7085-0/minicluster-data/master-0/wal/instance
I20260928 02:15:40.053728 7085 external_mini_cluster.cc:1428] Setting key ec4d553bef5247c93c7158b46711ebfb
I20260928 02:15:40.060352 24302 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b1442e6363434cd2ae8998baac5feb46: Bootstrap starting.
I20260928 02:15:40.063827 24302 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b1442e6363434cd2ae8998baac5feb46: Neither blocks nor log segments found. Creating new log.
I20260928 02:15:40.065022 24302 log.cc:824] T 00000000000000000000000000000000 P b1442e6363434cd2ae8998baac5feb46: Log is configured to *not* fsync() on all Append() calls
I20260928 02:15:40.068069 24302 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b1442e6363434cd2ae8998baac5feb46: No bootstrap required, opened a new log
Sep 28 02:15:40 dist-test-slave-86gv krb5kdc[24222](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561739, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.235.126@KRBTEST.COM
I20260928 02:15:40.076866 24302 raft_consensus.cc:348] T 00000000000000000000000000000000 P b1442e6363434cd2ae8998baac5feb46 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b1442e6363434cd2ae8998baac5feb46" member_type: VOTER last_known_addr { host: "127.6.235.126" port: 46297 } }
I20260928 02:15:40.077474 24302 raft_consensus.cc:374] T 00000000000000000000000000000000 P b1442e6363434cd2ae8998baac5feb46 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:15:40.077605 24302 raft_consensus.cc:729] T 00000000000000000000000000000000 P b1442e6363434cd2ae8998baac5feb46 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b1442e6363434cd2ae8998baac5feb46, State: Initialized, Role: FOLLOWER
I20260928 02:15:40.078235 24302 consensus_queue.cc:260] T 00000000000000000000000000000000 P b1442e6363434cd2ae8998baac5feb46 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b1442e6363434cd2ae8998baac5feb46" member_type: VOTER last_known_addr { host: "127.6.235.126" port: 46297 } }
I20260928 02:15:40.078472 24302 raft_consensus.cc:388] T 00000000000000000000000000000000 P b1442e6363434cd2ae8998baac5feb46 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 02:15:40.078625 24305 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:15:40.056073 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55210 (local address 127.6.235.126:46297)
0928 02:15:40.057074 (+ 1001us) server_negotiation.cc:206] Beginning negotiation
0928 02:15:40.057088 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0928 02:15:40.057173 (+ 85us) server_negotiation.cc:407] Connection header received
0928 02:15:40.059089 (+ 1916us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:15:40.059127 (+ 38us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:15:40.059564 (+ 437us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:15:40.059996 (+ 432us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:15:40.061609 (+ 1613us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:15:40.063002 (+ 1393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:15:40.064537 (+ 1535us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:15:40.065400 (+ 863us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:15:40.065625 (+ 225us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:15:40.070272 (+ 4647us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:15:40.070281 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:15:40.070298 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:15:40.070344 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:15:40.073831 (+ 3487us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:15:40.074644 (+ 813us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:15:40.074651 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:15:40.074657 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:15:40.074820 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:15:40.075241 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:15:40.075245 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:15:40.075251 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:15:40.075871 (+ 620us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:15:40.076217 (+ 346us) server_negotiation.cc:1035] Waiting for connection context
0928 02:15:40.076709 (+ 492us) server_negotiation.cc:299] Negotiation successful
0928 02:15:40.077228 (+ 519us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":699,"thread_start_us":486,"threads_started":1}
I20260928 02:15:40.078714 24302 raft_consensus.cc:482] T 00000000000000000000000000000000 P b1442e6363434cd2ae8998baac5feb46 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 02:15:40.079118 24302 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b1442e6363434cd2ae8998baac5feb46 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:15:40.081727 24302 raft_consensus.cc:504] T 00000000000000000000000000000000 P b1442e6363434cd2ae8998baac5feb46 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b1442e6363434cd2ae8998baac5feb46" member_type: VOTER last_known_addr { host: "127.6.235.126" port: 46297 } }
I20260928 02:15:40.082249 24302 leader_election.cc:302] T 00000000000000000000000000000000 P b1442e6363434cd2ae8998baac5feb46 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b1442e6363434cd2ae8998baac5feb46; no voters:
I20260928 02:15:40.082953 24302 leader_election.cc:288] T 00000000000000000000000000000000 P b1442e6363434cd2ae8998baac5feb46 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 02:15:40.083164 24307 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b1442e6363434cd2ae8998baac5feb46 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:15:40.084578 24307 raft_consensus.cc:686] T 00000000000000000000000000000000 P b1442e6363434cd2ae8998baac5feb46 [term 1 LEADER]: Becoming Leader. State: Replica: b1442e6363434cd2ae8998baac5feb46, State: Running, Role: LEADER
I20260928 02:15:40.085210 24302 sys_catalog.cc:564] T 00000000000000000000000000000000 P b1442e6363434cd2ae8998baac5feb46 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 02:15:40.085166 24307 consensus_queue.cc:237] T 00000000000000000000000000000000 P b1442e6363434cd2ae8998baac5feb46 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b1442e6363434cd2ae8998baac5feb46" member_type: VOTER last_known_addr { host: "127.6.235.126" port: 46297 } }
I20260928 02:15:40.088158 24308 sys_catalog.cc:455] T 00000000000000000000000000000000 P b1442e6363434cd2ae8998baac5feb46 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b1442e6363434cd2ae8998baac5feb46" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b1442e6363434cd2ae8998baac5feb46" member_type: VOTER last_known_addr { host: "127.6.235.126" port: 46297 } } }
I20260928 02:15:40.088299 24309 sys_catalog.cc:455] T 00000000000000000000000000000000 P b1442e6363434cd2ae8998baac5feb46 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b1442e6363434cd2ae8998baac5feb46. Latest consensus state: current_term: 1 leader_uuid: "b1442e6363434cd2ae8998baac5feb46" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b1442e6363434cd2ae8998baac5feb46" member_type: VOTER last_known_addr { host: "127.6.235.126" port: 46297 } } }
I20260928 02:15:40.088589 24308 sys_catalog.cc:458] T 00000000000000000000000000000000 P b1442e6363434cd2ae8998baac5feb46 [sys.catalog]: This master's current role is: LEADER
I20260928 02:15:40.088682 24309 sys_catalog.cc:458] T 00000000000000000000000000000000 P b1442e6363434cd2ae8998baac5feb46 [sys.catalog]: This master's current role is: LEADER
I20260928 02:15:40.091836 24314 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 02:15:40.097676 24314 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 02:15:40.107823 24314 catalog_manager.cc:1347] Generated new cluster ID: d6465c3de9f04af79c1a9cc4c9b06127
I20260928 02:15:40.107962 24314 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 02:15:40.131572 24314 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 02:15:40.133713 24314 catalog_manager.cc:1504] Loading token signing keys...
I20260928 02:15:40.141850 24314 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b1442e6363434cd2ae8998baac5feb46: Generated new TSK 0
I20260928 02:15:40.143118 24314 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.6.235.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.235.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.235.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790561432251112-7085-0/krb5kdc/kudu_127.6.235.65.keytab.
Entry for principal kudu/127.6.235.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790561432251112-7085-0/krb5kdc/kudu_127.6.235.65.keytab.
I20260928 02:15:40.221952 7085 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTlaUuc/build/asan/bin/kudu
/tmp/dist-test-taskTlaUuc/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790561432251112-7085-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790561432251112-7085-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790561432251112-7085-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790561432251112-7085-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.235.65:0
--local_ip_for_outbound_sockets=127.6.235.65
--webserver_interface=127.6.235.65
--webserver_port=0
--tserver_master_addrs=127.6.235.126:46297
--keytab_file=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790561432251112-7085-0/krb5kdc/kudu_127.6.235.65.keytab
--principal=kudu/127.6.235.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.235.84:39007
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790561432251112-7085-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790561432251112-7085-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790561432251112-7085-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:15:40.585110 24328 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:15:40.585510 24328 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:15:40.585718 24328 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:15:40.593192 24328 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:15:40.593374 24328 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:15:40.593580 24328 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.235.65
I20260928 02:15:40.605017 24328 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.235.84:39007
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790561432251112-7085-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790561432251112-7085-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790561432251112-7085-0/krb5kdc/kudu_127.6.235.65.keytab
--principal=kudu/127.6.235.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.235.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790561432251112-7085-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.6.235.65
--webserver_port=0
--tserver_master_addrs=127.6.235.126:46297
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.235.65
--log_dir=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790561432251112-7085-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 00:51:59 UTC on ccycloud.quasar-axgkwn.root.comops.site
build id 4676464
ASAN enabled
I20260928 02:15:40.606838 24328 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:15:40.608290 24328 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:15:40.623031 24333 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:15:40.623023 24336 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:15:40.623008 24334 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:15:40.623821 24328 server_base.cc:1034] running on GCE node
I20260928 02:15:40.624521 24328 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:15:40.625648 24328 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:15:40.626857 24328 hybrid_clock.cc:648] HybridClock initialized: now 1790561740626806 us; error 42 us; skew 500 ppm
I20260928 02:15:40.627296 24328 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 02:15:40 dist-test-slave-86gv krb5kdc[24222](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561740, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.235.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:15:40.630934 24328 init.cc:377] Logged in from keytab as kudu/127.6.235.65@KRBTEST.COM (short username kudu)
I20260928 02:15:40.632735 24328 webserver.cc:460] Webserver started at http://127.6.235.65:44377/ using document root <none> and password file <none>
I20260928 02:15:40.633776 24328 fs_manager.cc:359] Metadata directory not provided
I20260928 02:15:40.633960 24328 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:15:40.634341 24328 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:15:40.637977 24328 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790561432251112-7085-0/minicluster-data/ts-0/data/instance:
uuid: "124e240a84ba4220b0ddd48b758ab9f8"
format_stamp: "Formatted at 2026-09-28 02:15:40 on dist-test-slave-86gv"
server_key: "6a8a8b6658492af9b80c9689354ebb90"
server_key_iv: "9a76a63038916342e11c59dbe287c347"
server_key_version: "encryptionkey@0"
I20260928 02:15:40.638952 24328 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790561432251112-7085-0/minicluster-data/ts-0/wal/instance:
uuid: "124e240a84ba4220b0ddd48b758ab9f8"
format_stamp: "Formatted at 2026-09-28 02:15:40 on dist-test-slave-86gv"
server_key: "6a8a8b6658492af9b80c9689354ebb90"
server_key_iv: "9a76a63038916342e11c59dbe287c347"
server_key_version: "encryptionkey@0"
I20260928 02:15:40.646136 24328 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.002s sys 0.005s
I20260928 02:15:40.650925 24343 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:15:40.653064 24328 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.006s sys 0.000s
I20260928 02:15:40.653288 24328 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790561432251112-7085-0/minicluster-data/ts-0/data,/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790561432251112-7085-0/minicluster-data/ts-0/wal
uuid: "124e240a84ba4220b0ddd48b758ab9f8"
format_stamp: "Formatted at 2026-09-28 02:15:40 on dist-test-slave-86gv"
server_key: "6a8a8b6658492af9b80c9689354ebb90"
server_key_iv: "9a76a63038916342e11c59dbe287c347"
server_key_version: "encryptionkey@0"
I20260928 02:15:40.653540 24328 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790561432251112-7085-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790561432251112-7085-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790561432251112-7085-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:15:40.683477 24328 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:15:40.689399 24328 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:15:40.689807 24328 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:15:40.690924 24328 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:15:40.692591 24328 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:15:40.692706 24328 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:15:40.692868 24328 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:15:40.692973 24328 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.001s sys 0.000s
I20260928 02:15:40.728806 24328 rpc_server.cc:307] RPC server started. Bound to: 127.6.235.65:43853
I20260928 02:15:40.728920 24455 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.235.65:43853 every 8 connection(s)
I20260928 02:15:40.730223 24328 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790561432251112-7085-0/minicluster-data/ts-0/data/info.pb
I20260928 02:15:40.733310 7085 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTlaUuc/build/asan/bin/kudu as pid 24328
I20260928 02:15:40.733553 7085 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790561432251112-7085-0/minicluster-data/ts-0/wal/instance
I20260928 02:15:40.734352 7085 external_mini_cluster.cc:1428] Setting key 40a0a14c726300d39226bca31f6491ba
Sep 28 02:15:40 dist-test-slave-86gv krb5kdc[24222](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561740, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.235.65@KRBTEST.COM for kudu/127.6.235.126@KRBTEST.COM
I20260928 02:15:40.756515 24458 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:15:40.733281 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.235.65:55619 (local address 127.6.235.126:46297)
0928 02:15:40.735945 (+ 2664us) server_negotiation.cc:206] Beginning negotiation
0928 02:15:40.735951 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 02:15:40.735974 (+ 23us) server_negotiation.cc:407] Connection header received
0928 02:15:40.738223 (+ 2249us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:15:40.738227 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:15:40.738301 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:15:40.738402 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:15:40.740766 (+ 2364us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:15:40.741806 (+ 1040us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:15:40.744577 (+ 2771us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:15:40.745116 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:15:40.745301 (+ 185us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:15:40.749759 (+ 4458us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:15:40.749765 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:15:40.749768 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:15:40.749807 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:15:40.752721 (+ 2914us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:15:40.753924 (+ 1203us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:15:40.753929 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:15:40.753931 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:15:40.754063 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:15:40.754641 (+ 578us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:15:40.754645 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:15:40.754646 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:15:40.754917 (+ 271us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:15:40.755028 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0928 02:15:40.756061 (+ 1033us) server_negotiation.cc:299] Negotiation successful
0928 02:15:40.756262 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2547,"thread_start_us":2443,"threads_started":1}
I20260928 02:15:40.758376 24459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:15:40.734123 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.235.126:46297 (local address 127.6.235.65:55619)
0928 02:15:40.735273 (+ 1150us) negotiation.cc:107] Waiting for socket to connect
0928 02:15:40.735339 (+ 66us) client_negotiation.cc:174] Beginning negotiation
0928 02:15:40.737857 (+ 2518us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:15:40.738747 (+ 890us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:15:40.738764 (+ 17us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:15:40.739654 (+ 890us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:15:40.740548 (+ 894us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:15:40.740576 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:15:40.742018 (+ 1442us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:15:40.742026 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:15:40.744356 (+ 2330us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:15:40.744379 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:15:40.745441 (+ 1062us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:15:40.745451 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:15:40.745809 (+ 358us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:15:40.746625 (+ 816us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:15:40.746650 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:15:40.749450 (+ 2800us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:15:40.752982 (+ 3532us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:15:40.752996 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:15:40.753021 (+ 25us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:15:40.753780 (+ 759us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:15:40.754208 (+ 428us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:15:40.754216 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:15:40.754223 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:15:40.754510 (+ 287us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:15:40.755056 (+ 546us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:15:40.755068 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:15:40.755607 (+ 539us) client_negotiation.cc:769] Sending connection context
0928 02:15:40.755937 (+ 330us) client_negotiation.cc:240] Negotiation successful
0928 02:15:40.756577 (+ 640us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":706,"thread_start_us":410,"threads_started":1}
I20260928 02:15:40.760250 24456 heartbeater.cc:347] Connected to a master server at 127.6.235.126:46297
I20260928 02:15:40.760636 24456 heartbeater.cc:464] Registering TS with master...
I20260928 02:15:40.761654 24456 heartbeater.cc:511] Master 127.6.235.126:46297 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.6.235.66@KRBTEST.COM; defaulting to no policy
I20260928 02:15:40.764500 24267 ts_manager.cc:194] Registered new tserver with Master: 124e240a84ba4220b0ddd48b758ab9f8 (127.6.235.65:43853)
I20260928 02:15:40.766331 24267 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.235.65@KRBTEST.COM'} at 127.6.235.65:55619
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.235.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.235.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790561432251112-7085-0/krb5kdc/kudu_127.6.235.66.keytab.
Entry for principal kudu/127.6.235.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790561432251112-7085-0/krb5kdc/kudu_127.6.235.66.keytab.
I20260928 02:15:40.802044 7085 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTlaUuc/build/asan/bin/kudu
/tmp/dist-test-taskTlaUuc/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790561432251112-7085-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790561432251112-7085-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790561432251112-7085-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790561432251112-7085-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.235.66:0
--local_ip_for_outbound_sockets=127.6.235.66
--webserver_interface=127.6.235.66
--webserver_port=0
--tserver_master_addrs=127.6.235.126:46297
--keytab_file=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790561432251112-7085-0/krb5kdc/kudu_127.6.235.66.keytab
--principal=kudu/127.6.235.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.235.84:39007
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790561432251112-7085-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790561432251112-7085-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790561432251112-7085-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:15:41.175665 24462 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:15:41.176039 24462 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:15:41.176218 24462 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:15:41.183984 24462 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:15:41.184196 24462 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:15:41.184381 24462 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.235.66
I20260928 02:15:41.195149 24462 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.235.84:39007
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790561432251112-7085-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790561432251112-7085-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790561432251112-7085-0/krb5kdc/kudu_127.6.235.66.keytab
--principal=kudu/127.6.235.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.235.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790561432251112-7085-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.6.235.66
--webserver_port=0
--tserver_master_addrs=127.6.235.126:46297
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.235.66
--log_dir=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790561432251112-7085-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 00:51:59 UTC on ccycloud.quasar-axgkwn.root.comops.site
build id 4676464
ASAN enabled
I20260928 02:15:41.196767 24462 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:15:41.198365 24462 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:15:41.214289 24467 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:15:41.214337 24470 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:15:41.214360 24468 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:15:41.215363 24462 server_base.cc:1034] running on GCE node
I20260928 02:15:41.216095 24462 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:15:41.217332 24462 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:15:41.218585 24462 hybrid_clock.cc:648] HybridClock initialized: now 1790561741218527 us; error 63 us; skew 500 ppm
I20260928 02:15:41.218976 24462 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 02:15:41 dist-test-slave-86gv krb5kdc[24222](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561741, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.235.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:15:41.222709 24462 init.cc:377] Logged in from keytab as kudu/127.6.235.66@KRBTEST.COM (short username kudu)
I20260928 02:15:41.224344 24462 webserver.cc:460] Webserver started at http://127.6.235.66:35527/ using document root <none> and password file <none>
I20260928 02:15:41.225180 24462 fs_manager.cc:359] Metadata directory not provided
I20260928 02:15:41.225296 24462 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:15:41.225575 24462 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:15:41.228649 24462 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790561432251112-7085-0/minicluster-data/ts-1/data/instance:
uuid: "ef2b621fcd434003b7beacd3ce270895"
format_stamp: "Formatted at 2026-09-28 02:15:41 on dist-test-slave-86gv"
server_key: "05dda3aca06cf04a7c50456e34021e17"
server_key_iv: "aee1aa00bfd888c6d02faf98ff337fa9"
server_key_version: "encryptionkey@0"
I20260928 02:15:41.229578 24462 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790561432251112-7085-0/minicluster-data/ts-1/wal/instance:
uuid: "ef2b621fcd434003b7beacd3ce270895"
format_stamp: "Formatted at 2026-09-28 02:15:41 on dist-test-slave-86gv"
server_key: "05dda3aca06cf04a7c50456e34021e17"
server_key_iv: "aee1aa00bfd888c6d02faf98ff337fa9"
server_key_version: "encryptionkey@0"
I20260928 02:15:41.235575 24462 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260928 02:15:41.239516 24477 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:15:41.241232 24462 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 02:15:41.241431 24462 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790561432251112-7085-0/minicluster-data/ts-1/data,/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790561432251112-7085-0/minicluster-data/ts-1/wal
uuid: "ef2b621fcd434003b7beacd3ce270895"
format_stamp: "Formatted at 2026-09-28 02:15:41 on dist-test-slave-86gv"
server_key: "05dda3aca06cf04a7c50456e34021e17"
server_key_iv: "aee1aa00bfd888c6d02faf98ff337fa9"
server_key_version: "encryptionkey@0"
I20260928 02:15:41.241675 24462 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790561432251112-7085-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790561432251112-7085-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790561432251112-7085-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:15:41.258425 24462 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:15:41.263193 24462 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:15:41.263520 24462 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:15:41.264578 24462 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:15:41.266067 24462 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:15:41.266142 24462 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:15:41.266301 24462 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:15:41.266356 24462 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:15:41.303181 24462 rpc_server.cc:307] RPC server started. Bound to: 127.6.235.66:46763
I20260928 02:15:41.303239 24589 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.235.66:46763 every 8 connection(s)
I20260928 02:15:41.304625 24462 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790561432251112-7085-0/minicluster-data/ts-1/data/info.pb
I20260928 02:15:41.311614 7085 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTlaUuc/build/asan/bin/kudu as pid 24462
I20260928 02:15:41.311822 7085 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790561432251112-7085-0/minicluster-data/ts-1/wal/instance
I20260928 02:15:41.312384 7085 external_mini_cluster.cc:1428] Setting key 2ff789868a46da60567a6f441e28343d
Sep 28 02:15:41 dist-test-slave-86gv krb5kdc[24222](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561741, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.235.66@KRBTEST.COM for kudu/127.6.235.126@KRBTEST.COM
I20260928 02:15:41.333832 24592 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:15:41.307495 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.235.66:37459 (local address 127.6.235.126:46297)
0928 02:15:41.308013 (+ 518us) server_negotiation.cc:206] Beginning negotiation
0928 02:15:41.308019 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 02:15:41.309236 (+ 1217us) server_negotiation.cc:407] Connection header received
0928 02:15:41.311189 (+ 1953us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:15:41.311194 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:15:41.311273 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:15:41.311384 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:15:41.313515 (+ 2131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:15:41.314422 (+ 907us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:15:41.319658 (+ 5236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:15:41.320143 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:15:41.320270 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:15:41.325227 (+ 4957us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:15:41.325237 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:15:41.325246 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:15:41.325300 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:15:41.329407 (+ 4107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:15:41.330453 (+ 1046us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:15:41.330462 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:15:41.330469 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:15:41.330651 (+ 182us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:15:41.331375 (+ 724us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:15:41.331384 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:15:41.331390 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:15:41.331873 (+ 483us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:15:41.332039 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0928 02:15:41.333029 (+ 990us) server_negotiation.cc:299] Negotiation successful
0928 02:15:41.333308 (+ 279us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":388,"thread_start_us":252,"threads_started":1}
I20260928 02:15:41.334779 24593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:15:41.308111 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.235.126:46297 (local address 127.6.235.66:37459)
0928 02:15:41.308870 (+ 759us) negotiation.cc:107] Waiting for socket to connect
0928 02:15:41.308998 (+ 128us) client_negotiation.cc:174] Beginning negotiation
0928 02:15:41.310883 (+ 1885us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:15:41.311712 (+ 829us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:15:41.311726 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:15:41.312548 (+ 822us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:15:41.313343 (+ 795us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:15:41.313363 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:15:41.317091 (+ 3728us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:15:41.317100 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:15:41.319419 (+ 2319us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:15:41.319445 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:15:41.320435 (+ 990us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:15:41.320445 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:15:41.320802 (+ 357us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:15:41.321631 (+ 829us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:15:41.321669 (+ 38us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:15:41.324815 (+ 3146us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:15:41.329653 (+ 4838us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:15:41.329663 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:15:41.329681 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:15:41.330275 (+ 594us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:15:41.330827 (+ 552us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:15:41.330831 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:15:41.330834 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:15:41.331211 (+ 377us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:15:41.332118 (+ 907us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:15:41.332125 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:15:41.332544 (+ 419us) client_negotiation.cc:769] Sending connection context
0928 02:15:41.332843 (+ 299us) client_negotiation.cc:240] Negotiation successful
0928 02:15:41.333479 (+ 636us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":512,"thread_start_us":296,"threads_started":1}
I20260928 02:15:41.336565 24590 heartbeater.cc:347] Connected to a master server at 127.6.235.126:46297
I20260928 02:15:41.336990 24590 heartbeater.cc:464] Registering TS with master...
I20260928 02:15:41.338034 24590 heartbeater.cc:511] Master 127.6.235.126:46297 requested a full tablet report, sending...
I20260928 02:15:41.340235 24267 ts_manager.cc:194] Registered new tserver with Master: ef2b621fcd434003b7beacd3ce270895 (127.6.235.66:46763)
I20260928 02:15:41.341280 24267 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.235.66@KRBTEST.COM'} at 127.6.235.66:37459
WARNING: no policy specified for kudu/127.6.235.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.235.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.235.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790561432251112-7085-0/krb5kdc/kudu_127.6.235.67.keytab.
Entry for principal kudu/127.6.235.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790561432251112-7085-0/krb5kdc/kudu_127.6.235.67.keytab.
I20260928 02:15:41.378376 7085 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTlaUuc/build/asan/bin/kudu
/tmp/dist-test-taskTlaUuc/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790561432251112-7085-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790561432251112-7085-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790561432251112-7085-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790561432251112-7085-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.235.67:0
--local_ip_for_outbound_sockets=127.6.235.67
--webserver_interface=127.6.235.67
--webserver_port=0
--tserver_master_addrs=127.6.235.126:46297
--keytab_file=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790561432251112-7085-0/krb5kdc/kudu_127.6.235.67.keytab
--principal=kudu/127.6.235.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.235.84:39007
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790561432251112-7085-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790561432251112-7085-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790561432251112-7085-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:15:41.747663 24596 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:15:41.748059 24596 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:15:41.748274 24596 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:15:41.756623 24596 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:15:41.756805 24596 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:15:41.757041 24596 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.235.67
I20260928 02:15:41.767334 24596 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.235.84:39007
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790561432251112-7085-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790561432251112-7085-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790561432251112-7085-0/krb5kdc/kudu_127.6.235.67.keytab
--principal=kudu/127.6.235.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.235.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790561432251112-7085-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.6.235.67
--webserver_port=0
--tserver_master_addrs=127.6.235.126:46297
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.235.67
--log_dir=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790561432251112-7085-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 00:51:59 UTC on ccycloud.quasar-axgkwn.root.comops.site
build id 4676464
ASAN enabled
I20260928 02:15:41.768626 24596 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:15:41.770130 24596 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260928 02:15:41.770246 24456 heartbeater.cc:503] Master 127.6.235.126:46297 was elected leader, sending a full tablet report...
W20260928 02:15:41.784643 24604 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:15:41.784668 24602 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:15:41.784720 24601 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:15:41.790910 24596 server_base.cc:1034] running on GCE node
I20260928 02:15:41.791615 24596 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:15:41.792809 24596 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:15:41.794077 24596 hybrid_clock.cc:648] HybridClock initialized: now 1790561741794033 us; error 34 us; skew 500 ppm
I20260928 02:15:41.794454 24596 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 02:15:41 dist-test-slave-86gv krb5kdc[24222](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561741, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.235.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:15:41.798096 24596 init.cc:377] Logged in from keytab as kudu/127.6.235.67@KRBTEST.COM (short username kudu)
I20260928 02:15:41.799764 24596 webserver.cc:460] Webserver started at http://127.6.235.67:37201/ using document root <none> and password file <none>
I20260928 02:15:41.800604 24596 fs_manager.cc:359] Metadata directory not provided
I20260928 02:15:41.800761 24596 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:15:41.801156 24596 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:15:41.804203 24596 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790561432251112-7085-0/minicluster-data/ts-2/data/instance:
uuid: "f4c7f61ad3c74677bbfc8c88c0b2fca8"
format_stamp: "Formatted at 2026-09-28 02:15:41 on dist-test-slave-86gv"
server_key: "2717393222a7b25f98d9fd601a654401"
server_key_iv: "461c3ffa1b4a63a58414cf96193f34ec"
server_key_version: "encryptionkey@0"
I20260928 02:15:41.805101 24596 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790561432251112-7085-0/minicluster-data/ts-2/wal/instance:
uuid: "f4c7f61ad3c74677bbfc8c88c0b2fca8"
format_stamp: "Formatted at 2026-09-28 02:15:41 on dist-test-slave-86gv"
server_key: "2717393222a7b25f98d9fd601a654401"
server_key_iv: "461c3ffa1b4a63a58414cf96193f34ec"
server_key_version: "encryptionkey@0"
I20260928 02:15:41.810760 24596 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260928 02:15:41.814952 24611 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:15:41.816515 24596 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.001s
I20260928 02:15:41.816747 24596 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790561432251112-7085-0/minicluster-data/ts-2/data,/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790561432251112-7085-0/minicluster-data/ts-2/wal
uuid: "f4c7f61ad3c74677bbfc8c88c0b2fca8"
format_stamp: "Formatted at 2026-09-28 02:15:41 on dist-test-slave-86gv"
server_key: "2717393222a7b25f98d9fd601a654401"
server_key_iv: "461c3ffa1b4a63a58414cf96193f34ec"
server_key_version: "encryptionkey@0"
I20260928 02:15:41.817052 24596 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790561432251112-7085-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790561432251112-7085-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790561432251112-7085-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:15:41.835983 24596 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:15:41.840883 24596 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:15:41.841285 24596 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:15:41.842566 24596 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:15:41.844044 24596 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:15:41.844162 24596 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:15:41.844324 24596 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:15:41.844403 24596 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:15:41.881450 24596 rpc_server.cc:307] RPC server started. Bound to: 127.6.235.67:39717
I20260928 02:15:41.881541 24723 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.235.67:39717 every 8 connection(s)
I20260928 02:15:41.882933 24596 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790561432251112-7085-0/minicluster-data/ts-2/data/info.pb
I20260928 02:15:41.886533 7085 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTlaUuc/build/asan/bin/kudu as pid 24596
I20260928 02:15:41.886778 7085 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790561432251112-7085-0/minicluster-data/ts-2/wal/instance
I20260928 02:15:41.887409 7085 external_mini_cluster.cc:1428] Setting key 0d3d1318088d9875b2f3d74a304f6e2b
Sep 28 02:15:41 dist-test-slave-86gv krb5kdc[24222](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561741, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.235.67@KRBTEST.COM for kudu/127.6.235.126@KRBTEST.COM
I20260928 02:15:41.907037 24726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:15:41.885932 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.235.67:40891 (local address 127.6.235.126:46297)
0928 02:15:41.886491 (+ 559us) server_negotiation.cc:206] Beginning negotiation
0928 02:15:41.886498 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 02:15:41.887743 (+ 1245us) server_negotiation.cc:407] Connection header received
0928 02:15:41.890320 (+ 2577us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:15:41.890325 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:15:41.890414 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:15:41.890555 (+ 141us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:15:41.892539 (+ 1984us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:15:41.893655 (+ 1116us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:15:41.895262 (+ 1607us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:15:41.895665 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:15:41.895761 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:15:41.899646 (+ 3885us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:15:41.899652 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:15:41.899655 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:15:41.899694 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:15:41.903310 (+ 3616us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:15:41.904346 (+ 1036us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:15:41.904350 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:15:41.904353 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:15:41.904535 (+ 182us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:15:41.905273 (+ 738us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:15:41.905278 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:15:41.905281 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:15:41.905641 (+ 360us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:15:41.905791 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0928 02:15:41.906542 (+ 751us) server_negotiation.cc:299] Negotiation successful
0928 02:15:41.906738 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":420,"thread_start_us":260,"threads_started":1}
I20260928 02:15:41.908114 24727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:15:41.886555 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.235.126:46297 (local address 127.6.235.67:40891)
0928 02:15:41.887549 (+ 994us) negotiation.cc:107] Waiting for socket to connect
0928 02:15:41.887610 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0928 02:15:41.890035 (+ 2425us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:15:41.890918 (+ 883us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:15:41.890930 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:15:41.891630 (+ 700us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:15:41.892352 (+ 722us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:15:41.892371 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:15:41.893808 (+ 1437us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:15:41.893812 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:15:41.895122 (+ 1310us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:15:41.895133 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:15:41.895899 (+ 766us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:15:41.895904 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:15:41.896130 (+ 226us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:15:41.897003 (+ 873us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:15:41.897110 (+ 107us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:15:41.899384 (+ 2274us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:15:41.903514 (+ 4130us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:15:41.903527 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:15:41.903550 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:15:41.904168 (+ 618us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:15:41.904691 (+ 523us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:15:41.904695 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:15:41.904698 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:15:41.904981 (+ 283us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:15:41.905774 (+ 793us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:15:41.905781 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:15:41.906161 (+ 380us) client_negotiation.cc:769] Sending connection context
0928 02:15:41.906405 (+ 244us) client_negotiation.cc:240] Negotiation successful
0928 02:15:41.906854 (+ 449us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":645,"thread_start_us":365,"threads_started":1}
I20260928 02:15:41.909761 24724 heartbeater.cc:347] Connected to a master server at 127.6.235.126:46297
I20260928 02:15:41.910091 24724 heartbeater.cc:464] Registering TS with master...
I20260928 02:15:41.910961 24724 heartbeater.cc:511] Master 127.6.235.126:46297 requested a full tablet report, sending...
I20260928 02:15:41.912503 24267 ts_manager.cc:194] Registered new tserver with Master: f4c7f61ad3c74677bbfc8c88c0b2fca8 (127.6.235.67:39717)
I20260928 02:15:41.913405 24267 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.235.67@KRBTEST.COM'} at 127.6.235.67:40891
I20260928 02:15:41.917060 7085 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 02:15:41.938946 24726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:15:41.922527 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45748 (local address 127.6.235.126:46297)
0928 02:15:41.922733 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0928 02:15:41.922737 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:15:41.922882 (+ 145us) server_negotiation.cc:407] Connection header received
0928 02:15:41.923113 (+ 231us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:15:41.923119 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:15:41.923240 (+ 121us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:15:41.923409 (+ 169us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:15:41.925869 (+ 2460us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:15:41.927261 (+ 1392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:15:41.929574 (+ 2313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:15:41.930266 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:15:41.930426 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:15:41.933148 (+ 2722us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:15:41.933153 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:15:41.933157 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:15:41.933197 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:15:41.935880 (+ 2683us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:15:41.936763 (+ 883us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:15:41.936768 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:15:41.936771 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:15:41.937005 (+ 234us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:15:41.937418 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:15:41.937422 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:15:41.937424 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:15:41.937808 (+ 384us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:15:41.938002 (+ 194us) server_negotiation.cc:1035] Waiting for connection context
0928 02:15:41.938366 (+ 364us) server_negotiation.cc:299] Negotiation successful
0928 02:15:41.938594 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260928 02:15:41.943403 24267 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45748:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 02:15:41.951202 24267 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 02:15:41.975807 24740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:15:41.966354 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37762 (local address 127.6.235.65:43853)
0928 02:15:41.967070 (+ 716us) server_negotiation.cc:206] Beginning negotiation
0928 02:15:41.967080 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 02:15:41.967112 (+ 32us) server_negotiation.cc:407] Connection header received
0928 02:15:41.967215 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:15:41.967223 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:15:41.967517 (+ 294us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:15:41.968584 (+ 1067us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:15:41.968822 (+ 238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:15:41.970349 (+ 1527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:15:41.973331 (+ 2982us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:15:41.974606 (+ 1275us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:15:41.974763 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:15:41.975012 (+ 249us) server_negotiation.cc:1035] Waiting for connection context
0928 02:15:41.975262 (+ 250us) server_negotiation.cc:299] Negotiation successful
0928 02:15:41.975444 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":533,"thread_start_us":372,"threads_started":1}
I20260928 02:15:41.975880 24738 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:15:41.965062 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.235.65:43853 (local address 127.0.0.1:37762)
0928 02:15:41.966027 (+ 965us) negotiation.cc:107] Waiting for socket to connect
0928 02:15:41.966075 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0928 02:15:41.966270 (+ 195us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:15:41.967767 (+ 1497us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:15:41.967775 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:15:41.967807 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:15:41.968348 (+ 541us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:15:41.968366 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:15:41.970470 (+ 2104us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:15:41.970477 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:15:41.973152 (+ 2675us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:15:41.973169 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:15:41.974818 (+ 1649us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:15:41.974824 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:15:41.975013 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:15:41.975064 (+ 51us) client_negotiation.cc:769] Sending connection context
0928 02:15:41.975203 (+ 139us) client_negotiation.cc:240] Negotiation successful
0928 02:15:41.975404 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":679,"mutex_wait_us":2,"thread_start_us":2035,"threads_started":1}
I20260928 02:15:41.977427 24737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:15:41.965065 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.235.67:39717 (local address 127.0.0.1:47604)
0928 02:15:41.967543 (+ 2478us) negotiation.cc:107] Waiting for socket to connect
0928 02:15:41.967570 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0928 02:15:41.967713 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:15:41.969894 (+ 2181us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:15:41.969901 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:15:41.969926 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:15:41.970362 (+ 436us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:15:41.970375 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:15:41.972409 (+ 2034us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:15:41.972415 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:15:41.974382 (+ 1967us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:15:41.974396 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:15:41.976470 (+ 2074us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:15:41.976476 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:15:41.976645 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:15:41.976676 (+ 31us) client_negotiation.cc:769] Sending connection context
0928 02:15:41.976783 (+ 107us) client_negotiation.cc:240] Negotiation successful
0928 02:15:41.976942 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2222,"mutex_wait_us":183,"thread_start_us":861,"threads_started":1}
I20260928 02:15:41.977785 24741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:15:41.967215 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.235.66:46763 (local address 127.0.0.1:48248)
0928 02:15:41.968322 (+ 1107us) negotiation.cc:107] Waiting for socket to connect
0928 02:15:41.968344 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0928 02:15:41.968448 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:15:41.969604 (+ 1156us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:15:41.969609 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:15:41.969635 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:15:41.970021 (+ 386us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:15:41.970036 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:15:41.971515 (+ 1479us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:15:41.971523 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:15:41.973803 (+ 2280us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:15:41.973817 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:15:41.976071 (+ 2254us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:15:41.976079 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:15:41.976930 (+ 851us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:15:41.976969 (+ 39us) client_negotiation.cc:769] Sending connection context
0928 02:15:41.977055 (+ 86us) client_negotiation.cc:240] Negotiation successful
0928 02:15:41.977203 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":979,"mutex_wait_us":1,"thread_start_us":828,"threads_started":1}
I20260928 02:15:41.978355 24739 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:15:41.965907 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47604 (local address 127.6.235.67:39717)
0928 02:15:41.969331 (+ 3424us) server_negotiation.cc:206] Beginning negotiation
0928 02:15:41.969340 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 02:15:41.969371 (+ 31us) server_negotiation.cc:407] Connection header received
0928 02:15:41.969460 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:15:41.969467 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:15:41.969693 (+ 226us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:15:41.969881 (+ 188us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:15:41.970574 (+ 693us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:15:41.972239 (+ 1665us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:15:41.974565 (+ 2326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:15:41.976287 (+ 1722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:15:41.976416 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:15:41.976636 (+ 220us) server_negotiation.cc:1035] Waiting for connection context
0928 02:15:41.977818 (+ 1182us) server_negotiation.cc:299] Negotiation successful
0928 02:15:41.977994 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3261,"thread_start_us":3178,"threads_started":1}
I20260928 02:15:41.978763 24742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:15:41.967707 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48248 (local address 127.6.235.66:46763)
0928 02:15:41.968984 (+ 1277us) server_negotiation.cc:206] Beginning negotiation
0928 02:15:41.968994 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 02:15:41.969024 (+ 30us) server_negotiation.cc:407] Connection header received
0928 02:15:41.969131 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:15:41.969139 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:15:41.969403 (+ 264us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:15:41.969601 (+ 198us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:15:41.970205 (+ 604us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:15:41.971376 (+ 1171us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:15:41.973987 (+ 2611us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:15:41.975468 (+ 1481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:15:41.975622 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:15:41.975813 (+ 191us) server_negotiation.cc:1035] Waiting for connection context
0928 02:15:41.978286 (+ 2473us) server_negotiation.cc:299] Negotiation successful
0928 02:15:41.978471 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1070,"thread_start_us":946,"threads_started":1}
I20260928 02:15:41.980175 24391 tablet_service.cc:1467] Processing CreateTablet for tablet b279e25e695b47e3897e53eaf8241089 (DEFAULT_TABLE table=test-table [id=73de1e1e5b484ad2a422d22cd00e8994]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:15:41.981832 24391 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b279e25e695b47e3897e53eaf8241089. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:15:41.982507 24659 tablet_service.cc:1467] Processing CreateTablet for tablet b279e25e695b47e3897e53eaf8241089 (DEFAULT_TABLE table=test-table [id=73de1e1e5b484ad2a422d22cd00e8994]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:15:41.982441 24525 tablet_service.cc:1467] Processing CreateTablet for tablet b279e25e695b47e3897e53eaf8241089 (DEFAULT_TABLE table=test-table [id=73de1e1e5b484ad2a422d22cd00e8994]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:15:41.983762 24659 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b279e25e695b47e3897e53eaf8241089. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:15:41.983884 24525 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b279e25e695b47e3897e53eaf8241089. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:15:41.996443 24743 tablet_bootstrap.cc:492] T b279e25e695b47e3897e53eaf8241089 P f4c7f61ad3c74677bbfc8c88c0b2fca8: Bootstrap starting.
I20260928 02:15:42.000694 24743 tablet_bootstrap.cc:654] T b279e25e695b47e3897e53eaf8241089 P f4c7f61ad3c74677bbfc8c88c0b2fca8: Neither blocks nor log segments found. Creating new log.
I20260928 02:15:42.002170 24743 log.cc:824] T b279e25e695b47e3897e53eaf8241089 P f4c7f61ad3c74677bbfc8c88c0b2fca8: Log is configured to *not* fsync() on all Append() calls
I20260928 02:15:42.005491 24743 tablet_bootstrap.cc:492] T b279e25e695b47e3897e53eaf8241089 P f4c7f61ad3c74677bbfc8c88c0b2fca8: No bootstrap required, opened a new log
I20260928 02:15:42.005816 24743 ts_tablet_manager.cc:1397] T b279e25e695b47e3897e53eaf8241089 P f4c7f61ad3c74677bbfc8c88c0b2fca8: Time spent bootstrapping tablet: real 0.010s user 0.008s sys 0.000s
I20260928 02:15:42.008086 24744 tablet_bootstrap.cc:492] T b279e25e695b47e3897e53eaf8241089 P 124e240a84ba4220b0ddd48b758ab9f8: Bootstrap starting.
I20260928 02:15:42.010458 24745 tablet_bootstrap.cc:492] T b279e25e695b47e3897e53eaf8241089 P ef2b621fcd434003b7beacd3ce270895: Bootstrap starting.
I20260928 02:15:42.012575 24744 tablet_bootstrap.cc:654] T b279e25e695b47e3897e53eaf8241089 P 124e240a84ba4220b0ddd48b758ab9f8: Neither blocks nor log segments found. Creating new log.
I20260928 02:15:42.013720 24745 tablet_bootstrap.cc:654] T b279e25e695b47e3897e53eaf8241089 P ef2b621fcd434003b7beacd3ce270895: Neither blocks nor log segments found. Creating new log.
I20260928 02:15:42.013996 24744 log.cc:824] T b279e25e695b47e3897e53eaf8241089 P 124e240a84ba4220b0ddd48b758ab9f8: Log is configured to *not* fsync() on all Append() calls
I20260928 02:15:42.014920 24745 log.cc:824] T b279e25e695b47e3897e53eaf8241089 P ef2b621fcd434003b7beacd3ce270895: Log is configured to *not* fsync() on all Append() calls
I20260928 02:15:42.016782 24743 raft_consensus.cc:348] T b279e25e695b47e3897e53eaf8241089 P f4c7f61ad3c74677bbfc8c88c0b2fca8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4c7f61ad3c74677bbfc8c88c0b2fca8" member_type: VOTER last_known_addr { host: "127.6.235.67" port: 39717 } } peers { permanent_uuid: "124e240a84ba4220b0ddd48b758ab9f8" member_type: VOTER last_known_addr { host: "127.6.235.65" port: 43853 } } peers { permanent_uuid: "ef2b621fcd434003b7beacd3ce270895" member_type: VOTER last_known_addr { host: "127.6.235.66" port: 46763 } }
I20260928 02:15:42.017340 24743 raft_consensus.cc:374] T b279e25e695b47e3897e53eaf8241089 P f4c7f61ad3c74677bbfc8c88c0b2fca8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:15:42.017450 24743 raft_consensus.cc:729] T b279e25e695b47e3897e53eaf8241089 P f4c7f61ad3c74677bbfc8c88c0b2fca8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f4c7f61ad3c74677bbfc8c88c0b2fca8, State: Initialized, Role: FOLLOWER
I20260928 02:15:42.017725 24744 tablet_bootstrap.cc:492] T b279e25e695b47e3897e53eaf8241089 P 124e240a84ba4220b0ddd48b758ab9f8: No bootstrap required, opened a new log
I20260928 02:15:42.018042 24744 ts_tablet_manager.cc:1397] T b279e25e695b47e3897e53eaf8241089 P 124e240a84ba4220b0ddd48b758ab9f8: Time spent bootstrapping tablet: real 0.015s user 0.005s sys 0.004s
I20260928 02:15:42.018307 24743 consensus_queue.cc:260] T b279e25e695b47e3897e53eaf8241089 P f4c7f61ad3c74677bbfc8c88c0b2fca8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4c7f61ad3c74677bbfc8c88c0b2fca8" member_type: VOTER last_known_addr { host: "127.6.235.67" port: 39717 } } peers { permanent_uuid: "124e240a84ba4220b0ddd48b758ab9f8" member_type: VOTER last_known_addr { host: "127.6.235.65" port: 43853 } } peers { permanent_uuid: "ef2b621fcd434003b7beacd3ce270895" member_type: VOTER last_known_addr { host: "127.6.235.66" port: 46763 } }
I20260928 02:15:42.019763 24724 heartbeater.cc:503] Master 127.6.235.126:46297 was elected leader, sending a full tablet report...
I20260928 02:15:42.020781 24743 ts_tablet_manager.cc:1428] T b279e25e695b47e3897e53eaf8241089 P f4c7f61ad3c74677bbfc8c88c0b2fca8: Time spent starting tablet: real 0.015s user 0.014s sys 0.000s
I20260928 02:15:42.023047 24745 tablet_bootstrap.cc:492] T b279e25e695b47e3897e53eaf8241089 P ef2b621fcd434003b7beacd3ce270895: No bootstrap required, opened a new log
I20260928 02:15:42.023381 24745 ts_tablet_manager.cc:1397] T b279e25e695b47e3897e53eaf8241089 P ef2b621fcd434003b7beacd3ce270895: Time spent bootstrapping tablet: real 0.013s user 0.004s sys 0.005s
I20260928 02:15:42.029551 24744 raft_consensus.cc:348] T b279e25e695b47e3897e53eaf8241089 P 124e240a84ba4220b0ddd48b758ab9f8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4c7f61ad3c74677bbfc8c88c0b2fca8" member_type: VOTER last_known_addr { host: "127.6.235.67" port: 39717 } } peers { permanent_uuid: "124e240a84ba4220b0ddd48b758ab9f8" member_type: VOTER last_known_addr { host: "127.6.235.65" port: 43853 } } peers { permanent_uuid: "ef2b621fcd434003b7beacd3ce270895" member_type: VOTER last_known_addr { host: "127.6.235.66" port: 46763 } }
I20260928 02:15:42.030062 24744 raft_consensus.cc:374] T b279e25e695b47e3897e53eaf8241089 P 124e240a84ba4220b0ddd48b758ab9f8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:15:42.030153 24744 raft_consensus.cc:729] T b279e25e695b47e3897e53eaf8241089 P 124e240a84ba4220b0ddd48b758ab9f8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 124e240a84ba4220b0ddd48b758ab9f8, State: Initialized, Role: FOLLOWER
I20260928 02:15:42.030892 24744 consensus_queue.cc:260] T b279e25e695b47e3897e53eaf8241089 P 124e240a84ba4220b0ddd48b758ab9f8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4c7f61ad3c74677bbfc8c88c0b2fca8" member_type: VOTER last_known_addr { host: "127.6.235.67" port: 39717 } } peers { permanent_uuid: "124e240a84ba4220b0ddd48b758ab9f8" member_type: VOTER last_known_addr { host: "127.6.235.65" port: 43853 } } peers { permanent_uuid: "ef2b621fcd434003b7beacd3ce270895" member_type: VOTER last_known_addr { host: "127.6.235.66" port: 46763 } }
I20260928 02:15:42.032846 24744 ts_tablet_manager.cc:1428] T b279e25e695b47e3897e53eaf8241089 P 124e240a84ba4220b0ddd48b758ab9f8: Time spent starting tablet: real 0.015s user 0.011s sys 0.005s
I20260928 02:15:42.034235 24745 raft_consensus.cc:348] T b279e25e695b47e3897e53eaf8241089 P ef2b621fcd434003b7beacd3ce270895 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4c7f61ad3c74677bbfc8c88c0b2fca8" member_type: VOTER last_known_addr { host: "127.6.235.67" port: 39717 } } peers { permanent_uuid: "124e240a84ba4220b0ddd48b758ab9f8" member_type: VOTER last_known_addr { host: "127.6.235.65" port: 43853 } } peers { permanent_uuid: "ef2b621fcd434003b7beacd3ce270895" member_type: VOTER last_known_addr { host: "127.6.235.66" port: 46763 } }
I20260928 02:15:42.034780 24745 raft_consensus.cc:374] T b279e25e695b47e3897e53eaf8241089 P ef2b621fcd434003b7beacd3ce270895 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:15:42.034893 24745 raft_consensus.cc:729] T b279e25e695b47e3897e53eaf8241089 P ef2b621fcd434003b7beacd3ce270895 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef2b621fcd434003b7beacd3ce270895, State: Initialized, Role: FOLLOWER
I20260928 02:15:42.035565 24745 consensus_queue.cc:260] T b279e25e695b47e3897e53eaf8241089 P ef2b621fcd434003b7beacd3ce270895 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4c7f61ad3c74677bbfc8c88c0b2fca8" member_type: VOTER last_known_addr { host: "127.6.235.67" port: 39717 } } peers { permanent_uuid: "124e240a84ba4220b0ddd48b758ab9f8" member_type: VOTER last_known_addr { host: "127.6.235.65" port: 43853 } } peers { permanent_uuid: "ef2b621fcd434003b7beacd3ce270895" member_type: VOTER last_known_addr { host: "127.6.235.66" port: 46763 } }
I20260928 02:15:42.037061 24590 heartbeater.cc:503] Master 127.6.235.126:46297 was elected leader, sending a full tablet report...
I20260928 02:15:42.037528 24745 ts_tablet_manager.cc:1428] T b279e25e695b47e3897e53eaf8241089 P ef2b621fcd434003b7beacd3ce270895: Time spent starting tablet: real 0.014s user 0.015s sys 0.000s
W20260928 02:15:42.057022 24591 tablet.cc:2369] T b279e25e695b47e3897e53eaf8241089 P ef2b621fcd434003b7beacd3ce270895: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 02:15:42.134665 24725 tablet.cc:2369] T b279e25e695b47e3897e53eaf8241089 P f4c7f61ad3c74677bbfc8c88c0b2fca8: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 02:15:42.138926 24749 raft_consensus.cc:482] T b279e25e695b47e3897e53eaf8241089 P f4c7f61ad3c74677bbfc8c88c0b2fca8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 02:15:42.139250 24749 raft_consensus.cc:504] T b279e25e695b47e3897e53eaf8241089 P f4c7f61ad3c74677bbfc8c88c0b2fca8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4c7f61ad3c74677bbfc8c88c0b2fca8" member_type: VOTER last_known_addr { host: "127.6.235.67" port: 39717 } } peers { permanent_uuid: "124e240a84ba4220b0ddd48b758ab9f8" member_type: VOTER last_known_addr { host: "127.6.235.65" port: 43853 } } peers { permanent_uuid: "ef2b621fcd434003b7beacd3ce270895" member_type: VOTER last_known_addr { host: "127.6.235.66" port: 46763 } }
I20260928 02:15:42.141089 24749 leader_election.cc:288] T b279e25e695b47e3897e53eaf8241089 P f4c7f61ad3c74677bbfc8c88c0b2fca8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 124e240a84ba4220b0ddd48b758ab9f8 (127.6.235.65:43853), ef2b621fcd434003b7beacd3ce270895 (127.6.235.66:46763)
I20260928 02:15:42.149011 24740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:15:42.141586 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.235.67:34883 (local address 127.6.235.65:43853)
0928 02:15:42.141797 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0928 02:15:42.141805 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 02:15:42.141826 (+ 21us) server_negotiation.cc:407] Connection header received
0928 02:15:42.142103 (+ 277us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:15:42.142109 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:15:42.142207 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:15:42.142309 (+ 102us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:15:42.142799 (+ 490us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:15:42.143829 (+ 1030us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:15:42.146445 (+ 2616us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:15:42.147746 (+ 1301us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:15:42.147927 (+ 181us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:15:42.148003 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0928 02:15:42.148574 (+ 571us) server_negotiation.cc:299] Negotiation successful
0928 02:15:42.148710 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":88}
I20260928 02:15:42.149015 24742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:15:42.141653 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.235.67:40325 (local address 127.6.235.66:46763)
0928 02:15:42.141903 (+ 250us) server_negotiation.cc:206] Beginning negotiation
0928 02:15:42.141923 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0928 02:15:42.142432 (+ 509us) server_negotiation.cc:407] Connection header received
0928 02:15:42.142536 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:15:42.142539 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:15:42.142595 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:15:42.142674 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:15:42.143168 (+ 494us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:15:42.144126 (+ 958us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:15:42.146491 (+ 2365us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:15:42.147787 (+ 1296us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:15:42.147932 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:15:42.148003 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0928 02:15:42.148574 (+ 571us) server_negotiation.cc:299] Negotiation successful
0928 02:15:42.148740 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87}
I20260928 02:15:42.149091 24727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:15:42.141348 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.235.65:43853 (local address 127.6.235.67:34883)
0928 02:15:42.141738 (+ 390us) negotiation.cc:107] Waiting for socket to connect
0928 02:15:42.141760 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0928 02:15:42.141897 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:15:42.142368 (+ 471us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:15:42.142375 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:15:42.142406 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:15:42.142617 (+ 211us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:15:42.142625 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:15:42.143980 (+ 1355us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:15:42.143984 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:15:42.146212 (+ 2228us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:15:42.146230 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:15:42.147985 (+ 1755us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:15:42.147993 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:15:42.148365 (+ 372us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:15:42.148428 (+ 63us) client_negotiation.cc:769] Sending connection context
0928 02:15:42.148529 (+ 101us) client_negotiation.cc:240] Negotiation successful
0928 02:15:42.148705 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":193,"mutex_wait_us":32}
I20260928 02:15:42.149142 24752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:15:42.141496 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.235.66:46763 (local address 127.6.235.67:40325)
0928 02:15:42.142306 (+ 810us) negotiation.cc:107] Waiting for socket to connect
0928 02:15:42.142325 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0928 02:15:42.142430 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:15:42.142732 (+ 302us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:15:42.142736 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:15:42.142764 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:15:42.143027 (+ 263us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:15:42.143034 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:15:42.144267 (+ 1233us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:15:42.144272 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:15:42.146348 (+ 2076us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:15:42.146361 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:15:42.148012 (+ 1651us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:15:42.148017 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:15:42.148365 (+ 348us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:15:42.148420 (+ 55us) client_negotiation.cc:769] Sending connection context
0928 02:15:42.148561 (+ 141us) client_negotiation.cc:240] Negotiation successful
0928 02:15:42.148704 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":673,"mutex_wait_us":217,"thread_start_us":482,"threads_started":1}
I20260928 02:15:42.150300 24545 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b279e25e695b47e3897e53eaf8241089" candidate_uuid: "f4c7f61ad3c74677bbfc8c88c0b2fca8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef2b621fcd434003b7beacd3ce270895" is_pre_election: true
I20260928 02:15:42.150482 24411 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b279e25e695b47e3897e53eaf8241089" candidate_uuid: "f4c7f61ad3c74677bbfc8c88c0b2fca8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "124e240a84ba4220b0ddd48b758ab9f8" is_pre_election: true
I20260928 02:15:42.150974 24545 raft_consensus.cc:2454] T b279e25e695b47e3897e53eaf8241089 P ef2b621fcd434003b7beacd3ce270895 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f4c7f61ad3c74677bbfc8c88c0b2fca8 in term 0.
I20260928 02:15:42.150990 24411 raft_consensus.cc:2454] T b279e25e695b47e3897e53eaf8241089 P 124e240a84ba4220b0ddd48b758ab9f8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f4c7f61ad3c74677bbfc8c88c0b2fca8 in term 0.
I20260928 02:15:42.151736 24613 leader_election.cc:302] T b279e25e695b47e3897e53eaf8241089 P f4c7f61ad3c74677bbfc8c88c0b2fca8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 124e240a84ba4220b0ddd48b758ab9f8, f4c7f61ad3c74677bbfc8c88c0b2fca8; no voters:
I20260928 02:15:42.152230 24749 raft_consensus.cc:2781] T b279e25e695b47e3897e53eaf8241089 P f4c7f61ad3c74677bbfc8c88c0b2fca8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 02:15:42.152390 24749 raft_consensus.cc:482] T b279e25e695b47e3897e53eaf8241089 P f4c7f61ad3c74677bbfc8c88c0b2fca8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 02:15:42.152486 24749 raft_consensus.cc:3037] T b279e25e695b47e3897e53eaf8241089 P f4c7f61ad3c74677bbfc8c88c0b2fca8 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:15:42.154729 24749 raft_consensus.cc:504] T b279e25e695b47e3897e53eaf8241089 P f4c7f61ad3c74677bbfc8c88c0b2fca8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4c7f61ad3c74677bbfc8c88c0b2fca8" member_type: VOTER last_known_addr { host: "127.6.235.67" port: 39717 } } peers { permanent_uuid: "124e240a84ba4220b0ddd48b758ab9f8" member_type: VOTER last_known_addr { host: "127.6.235.65" port: 43853 } } peers { permanent_uuid: "ef2b621fcd434003b7beacd3ce270895" member_type: VOTER last_known_addr { host: "127.6.235.66" port: 46763 } }
I20260928 02:15:42.155388 24749 leader_election.cc:288] T b279e25e695b47e3897e53eaf8241089 P f4c7f61ad3c74677bbfc8c88c0b2fca8 [CANDIDATE]: Term 1 election: Requested vote from peers 124e240a84ba4220b0ddd48b758ab9f8 (127.6.235.65:43853), ef2b621fcd434003b7beacd3ce270895 (127.6.235.66:46763)
I20260928 02:15:42.155910 24411 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b279e25e695b47e3897e53eaf8241089" candidate_uuid: "f4c7f61ad3c74677bbfc8c88c0b2fca8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "124e240a84ba4220b0ddd48b758ab9f8"
I20260928 02:15:42.155990 24545 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b279e25e695b47e3897e53eaf8241089" candidate_uuid: "f4c7f61ad3c74677bbfc8c88c0b2fca8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef2b621fcd434003b7beacd3ce270895"
I20260928 02:15:42.156113 24411 raft_consensus.cc:3037] T b279e25e695b47e3897e53eaf8241089 P 124e240a84ba4220b0ddd48b758ab9f8 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:15:42.156271 24545 raft_consensus.cc:3037] T b279e25e695b47e3897e53eaf8241089 P ef2b621fcd434003b7beacd3ce270895 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:15:42.158869 24411 raft_consensus.cc:2454] T b279e25e695b47e3897e53eaf8241089 P 124e240a84ba4220b0ddd48b758ab9f8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f4c7f61ad3c74677bbfc8c88c0b2fca8 in term 1.
I20260928 02:15:42.159027 24545 raft_consensus.cc:2454] T b279e25e695b47e3897e53eaf8241089 P ef2b621fcd434003b7beacd3ce270895 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f4c7f61ad3c74677bbfc8c88c0b2fca8 in term 1.
I20260928 02:15:42.159360 24613 leader_election.cc:302] T b279e25e695b47e3897e53eaf8241089 P f4c7f61ad3c74677bbfc8c88c0b2fca8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 124e240a84ba4220b0ddd48b758ab9f8, f4c7f61ad3c74677bbfc8c88c0b2fca8; no voters:
I20260928 02:15:42.159747 24749 raft_consensus.cc:2781] T b279e25e695b47e3897e53eaf8241089 P f4c7f61ad3c74677bbfc8c88c0b2fca8 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:15:42.160279 24749 raft_consensus.cc:686] T b279e25e695b47e3897e53eaf8241089 P f4c7f61ad3c74677bbfc8c88c0b2fca8 [term 1 LEADER]: Becoming Leader. State: Replica: f4c7f61ad3c74677bbfc8c88c0b2fca8, State: Running, Role: LEADER
I20260928 02:15:42.160782 24749 consensus_queue.cc:237] T b279e25e695b47e3897e53eaf8241089 P f4c7f61ad3c74677bbfc8c88c0b2fca8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4c7f61ad3c74677bbfc8c88c0b2fca8" member_type: VOTER last_known_addr { host: "127.6.235.67" port: 39717 } } peers { permanent_uuid: "124e240a84ba4220b0ddd48b758ab9f8" member_type: VOTER last_known_addr { host: "127.6.235.65" port: 43853 } } peers { permanent_uuid: "ef2b621fcd434003b7beacd3ce270895" member_type: VOTER last_known_addr { host: "127.6.235.66" port: 46763 } }
I20260928 02:15:42.166332 24267 catalog_manager.cc:5495] T b279e25e695b47e3897e53eaf8241089 P f4c7f61ad3c74677bbfc8c88c0b2fca8 reported cstate change: term changed from 0 to 1, leader changed from <none> to f4c7f61ad3c74677bbfc8c88c0b2fca8 (127.6.235.67). New cstate: current_term: 1 leader_uuid: "f4c7f61ad3c74677bbfc8c88c0b2fca8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f4c7f61ad3c74677bbfc8c88c0b2fca8" member_type: VOTER last_known_addr { host: "127.6.235.67" port: 39717 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "124e240a84ba4220b0ddd48b758ab9f8" member_type: VOTER last_known_addr { host: "127.6.235.65" port: 43853 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ef2b621fcd434003b7beacd3ce270895" member_type: VOTER last_known_addr { host: "127.6.235.66" port: 46763 } health_report { overall_health: UNKNOWN } } }
I20260928 02:15:42.223944 24739 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:15:42.217824 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47614 (local address 127.6.235.67:39717)
0928 02:15:42.218042 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0928 02:15:42.218048 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 02:15:42.218071 (+ 23us) server_negotiation.cc:407] Connection header received
0928 02:15:42.218204 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:15:42.218209 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:15:42.218298 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:15:42.218461 (+ 163us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 02:15:42.218847 (+ 386us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:15:42.219771 (+ 924us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:15:42.221511 (+ 1740us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:15:42.221946 (+ 435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:15:42.222041 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:15:42.222620 (+ 579us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 02:15:42.223255 (+ 635us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 02:15:42.223388 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0928 02:15:42.223503 (+ 115us) server_negotiation.cc:299] Negotiation successful
0928 02:15:42.223577 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260928 02:15:42.232817 24545 raft_consensus.cc:1261] T b279e25e695b47e3897e53eaf8241089 P ef2b621fcd434003b7beacd3ce270895 [term 1 FOLLOWER]: Refusing update from remote peer f4c7f61ad3c74677bbfc8c88c0b2fca8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 02:15:42.233043 24411 raft_consensus.cc:1261] T b279e25e695b47e3897e53eaf8241089 P 124e240a84ba4220b0ddd48b758ab9f8 [term 1 FOLLOWER]: Refusing update from remote peer f4c7f61ad3c74677bbfc8c88c0b2fca8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20260928 02:15:42.233182 24457 tablet.cc:2369] T b279e25e695b47e3897e53eaf8241089 P 124e240a84ba4220b0ddd48b758ab9f8: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 02:15:42.234112 24753 consensus_queue.cc:1035] T b279e25e695b47e3897e53eaf8241089 P f4c7f61ad3c74677bbfc8c88c0b2fca8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ef2b621fcd434003b7beacd3ce270895" member_type: VOTER last_known_addr { host: "127.6.235.66" port: 46763 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 02:15:42.234539 24749 consensus_queue.cc:1035] T b279e25e695b47e3897e53eaf8241089 P f4c7f61ad3c74677bbfc8c88c0b2fca8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "124e240a84ba4220b0ddd48b758ab9f8" member_type: VOTER last_known_addr { host: "127.6.235.65" port: 43853 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 02:15:42.246857 24756 mvcc.cc:205] Tried to move back new op lower bound from 7334140896175345664 to 7334140895895621632. Current Snapshot: MvccSnapshot[applied={T|T < 7334140896175345664}]
I20260928 02:15:42.263792 24267 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45748:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 02:15:42.264169 24267 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45748:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 02:15:42.268221 24267 catalog_manager.cc:5782] T 00000000000000000000000000000000 P b1442e6363434cd2ae8998baac5feb46: Sending DeleteTablet for 3 replicas of tablet b279e25e695b47e3897e53eaf8241089
I20260928 02:15:42.269680 24525 tablet_service.cc:1514] Processing DeleteTablet for tablet b279e25e695b47e3897e53eaf8241089 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:15:42 UTC) from {username='kudu', principal='kudu/127.6.235.126@KRBTEST.COM'} at 127.0.0.1:48248
I20260928 02:15:42.269635 24391 tablet_service.cc:1514] Processing DeleteTablet for tablet b279e25e695b47e3897e53eaf8241089 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:15:42 UTC) from {username='kudu', principal='kudu/127.6.235.126@KRBTEST.COM'} at 127.0.0.1:37762
I20260928 02:15:42.270043 24659 tablet_service.cc:1514] Processing DeleteTablet for tablet b279e25e695b47e3897e53eaf8241089 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:15:42 UTC) from {username='kudu', principal='kudu/127.6.235.126@KRBTEST.COM'} at 127.0.0.1:47604
I20260928 02:15:42.272284 7085 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskTlaUuc/build/asan/bin/kudu pid 24328
I20260928 02:15:42.282712 24766 tablet_replica.cc:330] T b279e25e695b47e3897e53eaf8241089 P 124e240a84ba4220b0ddd48b758ab9f8: stopping tablet replica
I20260928 02:15:42.283222 24767 tablet_replica.cc:330] T b279e25e695b47e3897e53eaf8241089 P f4c7f61ad3c74677bbfc8c88c0b2fca8: stopping tablet replica
I20260928 02:15:42.283318 24765 tablet_replica.cc:330] T b279e25e695b47e3897e53eaf8241089 P ef2b621fcd434003b7beacd3ce270895: stopping tablet replica
I20260928 02:15:42.283274 24766 raft_consensus.cc:2229] T b279e25e695b47e3897e53eaf8241089 P 124e240a84ba4220b0ddd48b758ab9f8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 02:15:42.283733 24765 raft_consensus.cc:2229] T b279e25e695b47e3897e53eaf8241089 P ef2b621fcd434003b7beacd3ce270895 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 02:15:42.283851 24766 raft_consensus.cc:2258] T b279e25e695b47e3897e53eaf8241089 P 124e240a84ba4220b0ddd48b758ab9f8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 02:15:42.283787 24767 raft_consensus.cc:2229] T b279e25e695b47e3897e53eaf8241089 P f4c7f61ad3c74677bbfc8c88c0b2fca8 [term 1 LEADER]: Raft consensus shutting down.
I20260928 02:15:42.284281 24765 raft_consensus.cc:2258] T b279e25e695b47e3897e53eaf8241089 P ef2b621fcd434003b7beacd3ce270895 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 02:15:42.284751 24767 raft_consensus.cc:2258] T b279e25e695b47e3897e53eaf8241089 P f4c7f61ad3c74677bbfc8c88c0b2fca8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 02:15:42.287206 24766 ts_tablet_manager.cc:1905] T b279e25e695b47e3897e53eaf8241089 P 124e240a84ba4220b0ddd48b758ab9f8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 02:15:42.289136 24765 ts_tablet_manager.cc:1905] T b279e25e695b47e3897e53eaf8241089 P ef2b621fcd434003b7beacd3ce270895: Deleting tablet data with delete state TABLET_DATA_DELETED
Sep 28 02:15:42 dist-test-slave-86gv krb5kdc[24222](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561739, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.235.65@KRBTEST.COM
I20260928 02:15:42.294104 24767 ts_tablet_manager.cc:1905] T b279e25e695b47e3897e53eaf8241089 P f4c7f61ad3c74677bbfc8c88c0b2fca8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 02:15:42.296521 24765 ts_tablet_manager.cc:1918] T b279e25e695b47e3897e53eaf8241089 P ef2b621fcd434003b7beacd3ce270895: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 02:15:42.296693 24765 log.cc:1196] T b279e25e695b47e3897e53eaf8241089 P ef2b621fcd434003b7beacd3ce270895: Deleting WAL directory at /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790561432251112-7085-0/minicluster-data/ts-1/wal/wals/b279e25e695b47e3897e53eaf8241089
I20260928 02:15:42.297374 24765 ts_tablet_manager.cc:1939] T b279e25e695b47e3897e53eaf8241089 P ef2b621fcd434003b7beacd3ce270895: Deleting consensus metadata
I20260928 02:15:42.299541 24251 catalog_manager.cc:4841] TS ef2b621fcd434003b7beacd3ce270895 (127.6.235.66:46763): tablet b279e25e695b47e3897e53eaf8241089 (table test-table [id=73de1e1e5b484ad2a422d22cd00e8994]) successfully deleted
I20260928 02:15:42.301987 24767 ts_tablet_manager.cc:1918] T b279e25e695b47e3897e53eaf8241089 P f4c7f61ad3c74677bbfc8c88c0b2fca8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 02:15:42.301973 24766 ts_tablet_manager.cc:1918] T b279e25e695b47e3897e53eaf8241089 P 124e240a84ba4220b0ddd48b758ab9f8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 02:15:42.302153 24766 log.cc:1196] T b279e25e695b47e3897e53eaf8241089 P 124e240a84ba4220b0ddd48b758ab9f8: Deleting WAL directory at /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790561432251112-7085-0/minicluster-data/ts-0/wal/wals/b279e25e695b47e3897e53eaf8241089
I20260928 02:15:42.302169 24767 log.cc:1196] T b279e25e695b47e3897e53eaf8241089 P f4c7f61ad3c74677bbfc8c88c0b2fca8: Deleting WAL directory at /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790561432251112-7085-0/minicluster-data/ts-2/wal/wals/b279e25e695b47e3897e53eaf8241089
I20260928 02:15:42.302738 24766 ts_tablet_manager.cc:1939] T b279e25e695b47e3897e53eaf8241089 P 124e240a84ba4220b0ddd48b758ab9f8: Deleting consensus metadata
I20260928 02:15:42.302867 24767 ts_tablet_manager.cc:1939] T b279e25e695b47e3897e53eaf8241089 P f4c7f61ad3c74677bbfc8c88c0b2fca8: Deleting consensus metadata
I20260928 02:15:42.304607 24252 catalog_manager.cc:4841] TS 124e240a84ba4220b0ddd48b758ab9f8 (127.6.235.65:43853): tablet b279e25e695b47e3897e53eaf8241089 (table test-table [id=73de1e1e5b484ad2a422d22cd00e8994]) successfully deleted
I20260928 02:15:42.304802 24251 catalog_manager.cc:4841] TS f4c7f61ad3c74677bbfc8c88c0b2fca8 (127.6.235.67:39717): tablet b279e25e695b47e3897e53eaf8241089 (table test-table [id=73de1e1e5b484ad2a422d22cd00e8994]) successfully deleted
I20260928 02:15:42.306167 24740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:15:42.274304 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37778 (local address 127.6.235.65:43853)
0928 02:15:42.274555 (+ 251us) server_negotiation.cc:206] Beginning negotiation
0928 02:15:42.274564 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 02:15:42.277523 (+ 2959us) server_negotiation.cc:407] Connection header received
0928 02:15:42.277877 (+ 354us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:15:42.277886 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:15:42.278148 (+ 262us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:15:42.278346 (+ 198us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:15:42.284987 (+ 6641us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:15:42.286065 (+ 1078us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:15:42.288131 (+ 2066us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:15:42.288842 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:15:42.289016 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:15:42.295233 (+ 6217us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:15:42.295252 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:15:42.295272 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:15:42.295323 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:15:42.300423 (+ 5100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:15:42.303107 (+ 2684us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:15:42.303117 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:15:42.303121 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:15:42.303325 (+ 204us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:15:42.303827 (+ 502us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:15:42.303833 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:15:42.303836 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:15:42.304325 (+ 489us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:15:42.304620 (+ 295us) server_negotiation.cc:1035] Waiting for connection context
0928 02:15:42.305519 (+ 899us) server_negotiation.cc:299] Negotiation successful
0928 02:15:42.305803 (+ 284us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260928 02:15:42.307969 24451 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 02:15:42.413076 7085 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTlaUuc/build/asan/bin/kudu with pid 24328
I20260928 02:15:42.434571 7085 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskTlaUuc/build/asan/bin/kudu pid 24462
Sep 28 02:15:42 dist-test-slave-86gv krb5kdc[24222](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561739, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.235.66@KRBTEST.COM
I20260928 02:15:42.453007 24742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:15:42.435583 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48262 (local address 127.6.235.66:46763)
0928 02:15:42.435873 (+ 290us) server_negotiation.cc:206] Beginning negotiation
0928 02:15:42.435880 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 02:15:42.435904 (+ 24us) server_negotiation.cc:407] Connection header received
0928 02:15:42.435976 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:15:42.435982 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:15:42.436081 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:15:42.436198 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:15:42.437738 (+ 1540us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:15:42.438595 (+ 857us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:15:42.440313 (+ 1718us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:15:42.440764 (+ 451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:15:42.440954 (+ 190us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:15:42.445906 (+ 4952us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:15:42.445924 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:15:42.445942 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:15:42.445990 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:15:42.449657 (+ 3667us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:15:42.450506 (+ 849us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:15:42.450514 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:15:42.450517 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:15:42.450663 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:15:42.451113 (+ 450us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:15:42.451123 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:15:42.451127 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:15:42.451553 (+ 426us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:15:42.451949 (+ 396us) server_negotiation.cc:1035] Waiting for connection context
0928 02:15:42.452292 (+ 343us) server_negotiation.cc:299] Negotiation successful
0928 02:15:42.452538 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":141}
I20260928 02:15:42.453761 24585 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 02:15:42.549614 7085 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTlaUuc/build/asan/bin/kudu with pid 24462
I20260928 02:15:42.572316 7085 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskTlaUuc/build/asan/bin/kudu pid 24596
Sep 28 02:15:42 dist-test-slave-86gv krb5kdc[24222](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561739, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.235.67@KRBTEST.COM
I20260928 02:15:42.591815 24739 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:15:42.573271 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47628 (local address 127.6.235.67:39717)
0928 02:15:42.573488 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0928 02:15:42.573495 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 02:15:42.573518 (+ 23us) server_negotiation.cc:407] Connection header received
0928 02:15:42.573590 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:15:42.573596 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:15:42.573732 (+ 136us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:15:42.573866 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:15:42.575247 (+ 1381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:15:42.576439 (+ 1192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:15:42.577924 (+ 1485us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:15:42.578656 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:15:42.578815 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:15:42.583838 (+ 5023us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:15:42.583863 (+ 25us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:15:42.583888 (+ 25us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:15:42.583946 (+ 58us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:15:42.588399 (+ 4453us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:15:42.589332 (+ 933us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:15:42.589345 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:15:42.589349 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:15:42.589561 (+ 212us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:15:42.589972 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:15:42.589977 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:15:42.589980 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:15:42.590500 (+ 520us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:15:42.590858 (+ 358us) server_negotiation.cc:1035] Waiting for connection context
0928 02:15:42.591139 (+ 281us) server_negotiation.cc:299] Negotiation successful
0928 02:15:42.591379 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260928 02:15:42.592844 24719 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 02:15:42.725991 7085 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTlaUuc/build/asan/bin/kudu with pid 24596
I20260928 02:15:42.746565 7085 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskTlaUuc/build/asan/bin/kudu pid 24235
I20260928 02:15:42.747884 24297 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 02:15:42.863155 7085 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTlaUuc/build/asan/bin/kudu with pid 24235
2026-09-28T02:15:42Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/0 (4698 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/1
2026-09-28T02:15:42Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T02:15:42Z Disabled control of system clock
I20260928 02:15:42.947865 7085 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTlaUuc/build/asan/bin/kudu
/tmp/dist-test-taskTlaUuc/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790561432251112-7085-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790561432251112-7085-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790561432251112-7085-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790561432251112-7085-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.235.126:38927
--webserver_interface=127.6.235.126
--webserver_port=0
--builtin_ntp_servers=127.6.235.84:46507
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.235.126:38927
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260928 02:15:43.306107 24776 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:15:43.306452 24776 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:15:43.306542 24776 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:15:43.314409 24776 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 02:15:43.314548 24776 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:15:43.314592 24776 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:15:43.314658 24776 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 02:15:43.314710 24776 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 02:15:43.324795 24776 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.235.84:46507
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790561432251112-7085-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790561432251112-7085-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.235.126:38927
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.235.126:38927
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790561432251112-7085-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.235.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790561432251112-7085-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 00:51:59 UTC on ccycloud.quasar-axgkwn.root.comops.site
build id 4676464
ASAN enabled
I20260928 02:15:43.326182 24776 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:15:43.328022 24776 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:15:43.341328 24781 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:15:43.341364 24784 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:15:43.341393 24782 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:15:43.342242 24776 server_base.cc:1034] running on GCE node
I20260928 02:15:43.343150 24776 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:15:43.344779 24776 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:15:43.346022 24776 hybrid_clock.cc:648] HybridClock initialized: now 1790561743345947 us; error 61 us; skew 500 ppm
I20260928 02:15:43.346477 24776 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260928 02:15:43.355294 24776 webserver.cc:460] Webserver started at http://127.6.235.126:43593/ using document root <none> and password file <none>
I20260928 02:15:43.356231 24776 fs_manager.cc:359] Metadata directory not provided
I20260928 02:15:43.356380 24776 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:15:43.356783 24776 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:15:43.360212 24776 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790561432251112-7085-0/minicluster-data/master-0/data/instance:
uuid: "5cb923917cdd4dd8bd6ebdd8b7634d16"
format_stamp: "Formatted at 2026-09-28 02:15:43 on dist-test-slave-86gv"
server_key: "977c9218a3981a11a2c9fa508e23e316"
server_key_iv: "7e78dac702be342a5c662005368cd469"
server_key_version: "encryptionkey@0"
I20260928 02:15:43.361290 24776 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790561432251112-7085-0/minicluster-data/master-0/wal/instance:
uuid: "5cb923917cdd4dd8bd6ebdd8b7634d16"
format_stamp: "Formatted at 2026-09-28 02:15:43 on dist-test-slave-86gv"
server_key: "977c9218a3981a11a2c9fa508e23e316"
server_key_iv: "7e78dac702be342a5c662005368cd469"
server_key_version: "encryptionkey@0"
I20260928 02:15:43.367736 24776 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260928 02:15:43.371964 24790 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:15:43.373898 24776 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260928 02:15:43.374184 24776 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790561432251112-7085-0/minicluster-data/master-0/data,/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790561432251112-7085-0/minicluster-data/master-0/wal
uuid: "5cb923917cdd4dd8bd6ebdd8b7634d16"
format_stamp: "Formatted at 2026-09-28 02:15:43 on dist-test-slave-86gv"
server_key: "977c9218a3981a11a2c9fa508e23e316"
server_key_iv: "7e78dac702be342a5c662005368cd469"
server_key_version: "encryptionkey@0"
I20260928 02:15:43.374490 24776 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790561432251112-7085-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790561432251112-7085-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790561432251112-7085-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:15:43.389338 24776 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:15:43.390296 24776 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:15:43.390612 24776 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:15:43.411865 24776 rpc_server.cc:307] RPC server started. Bound to: 127.6.235.126:38927
I20260928 02:15:43.411888 24841 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.235.126:38927 every 8 connection(s)
I20260928 02:15:43.413484 24776 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790561432251112-7085-0/minicluster-data/master-0/data/info.pb
I20260928 02:15:43.416026 7085 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTlaUuc/build/asan/bin/kudu as pid 24776
I20260928 02:15:43.416240 7085 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790561432251112-7085-0/minicluster-data/master-0/wal/instance
I20260928 02:15:43.416723 7085 external_mini_cluster.cc:1428] Setting key bd56b83289b2303b88e3d07aa409c93c
I20260928 02:15:43.418180 24842 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:15:43.430997 24845 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:15:43.418800 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45508 (local address 127.6.235.126:38927)
0928 02:15:43.419904 (+ 1104us) server_negotiation.cc:206] Beginning negotiation
0928 02:15:43.419916 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0928 02:15:43.419961 (+ 45us) server_negotiation.cc:407] Connection header received
0928 02:15:43.421679 (+ 1718us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:15:43.421705 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:15:43.422239 (+ 534us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:15:43.422720 (+ 481us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:15:43.423354 (+ 634us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:15:43.424738 (+ 1384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:15:43.426244 (+ 1506us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:15:43.426951 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:15:43.427172 (+ 221us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:15:43.428438 (+ 1266us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:15:43.428445 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:15:43.428461 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 02:15:43.429169 (+ 708us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:15:43.429251 (+ 82us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 02:15:43.429274 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:15:43.429403 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0928 02:15:43.429633 (+ 230us) server_negotiation.cc:299] Negotiation successful
0928 02:15:43.429982 (+ 349us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":618,"thread_start_us":402,"threads_started":1}
I20260928 02:15:43.431051 24842 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5cb923917cdd4dd8bd6ebdd8b7634d16: Bootstrap starting.
I20260928 02:15:43.434916 24842 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5cb923917cdd4dd8bd6ebdd8b7634d16: Neither blocks nor log segments found. Creating new log.
I20260928 02:15:43.436062 24842 log.cc:824] T 00000000000000000000000000000000 P 5cb923917cdd4dd8bd6ebdd8b7634d16: Log is configured to *not* fsync() on all Append() calls
I20260928 02:15:43.439256 24842 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5cb923917cdd4dd8bd6ebdd8b7634d16: No bootstrap required, opened a new log
I20260928 02:15:43.447260 24842 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5cb923917cdd4dd8bd6ebdd8b7634d16 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5cb923917cdd4dd8bd6ebdd8b7634d16" member_type: VOTER last_known_addr { host: "127.6.235.126" port: 38927 } }
I20260928 02:15:43.447713 24842 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5cb923917cdd4dd8bd6ebdd8b7634d16 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:15:43.447825 24842 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5cb923917cdd4dd8bd6ebdd8b7634d16 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5cb923917cdd4dd8bd6ebdd8b7634d16, State: Initialized, Role: FOLLOWER
I20260928 02:15:43.448508 24842 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5cb923917cdd4dd8bd6ebdd8b7634d16 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5cb923917cdd4dd8bd6ebdd8b7634d16" member_type: VOTER last_known_addr { host: "127.6.235.126" port: 38927 } }
I20260928 02:15:43.448768 24842 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5cb923917cdd4dd8bd6ebdd8b7634d16 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 02:15:43.448935 24842 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5cb923917cdd4dd8bd6ebdd8b7634d16 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 02:15:43.449147 24842 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5cb923917cdd4dd8bd6ebdd8b7634d16 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:15:43.451812 24842 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5cb923917cdd4dd8bd6ebdd8b7634d16 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5cb923917cdd4dd8bd6ebdd8b7634d16" member_type: VOTER last_known_addr { host: "127.6.235.126" port: 38927 } }
I20260928 02:15:43.452364 24842 leader_election.cc:302] T 00000000000000000000000000000000 P 5cb923917cdd4dd8bd6ebdd8b7634d16 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5cb923917cdd4dd8bd6ebdd8b7634d16; no voters:
I20260928 02:15:43.453069 24842 leader_election.cc:288] T 00000000000000000000000000000000 P 5cb923917cdd4dd8bd6ebdd8b7634d16 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 02:15:43.453220 24847 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5cb923917cdd4dd8bd6ebdd8b7634d16 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:15:43.453946 24847 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5cb923917cdd4dd8bd6ebdd8b7634d16 [term 1 LEADER]: Becoming Leader. State: Replica: 5cb923917cdd4dd8bd6ebdd8b7634d16, State: Running, Role: LEADER
I20260928 02:15:43.454468 24847 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5cb923917cdd4dd8bd6ebdd8b7634d16 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5cb923917cdd4dd8bd6ebdd8b7634d16" member_type: VOTER last_known_addr { host: "127.6.235.126" port: 38927 } }
I20260928 02:15:43.455292 24842 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5cb923917cdd4dd8bd6ebdd8b7634d16 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 02:15:43.457553 24848 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5cb923917cdd4dd8bd6ebdd8b7634d16 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5cb923917cdd4dd8bd6ebdd8b7634d16" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5cb923917cdd4dd8bd6ebdd8b7634d16" member_type: VOTER last_known_addr { host: "127.6.235.126" port: 38927 } } }
I20260928 02:15:43.457578 24849 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5cb923917cdd4dd8bd6ebdd8b7634d16 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5cb923917cdd4dd8bd6ebdd8b7634d16. Latest consensus state: current_term: 1 leader_uuid: "5cb923917cdd4dd8bd6ebdd8b7634d16" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5cb923917cdd4dd8bd6ebdd8b7634d16" member_type: VOTER last_known_addr { host: "127.6.235.126" port: 38927 } } }
I20260928 02:15:43.459327 24848 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5cb923917cdd4dd8bd6ebdd8b7634d16 [sys.catalog]: This master's current role is: LEADER
I20260928 02:15:43.459375 24849 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5cb923917cdd4dd8bd6ebdd8b7634d16 [sys.catalog]: This master's current role is: LEADER
I20260928 02:15:43.461108 24854 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 02:15:43.467175 24854 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 02:15:43.477110 24854 catalog_manager.cc:1347] Generated new cluster ID: a1f2d56357674d35a9dc854752f46e51
I20260928 02:15:43.477239 24854 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 02:15:43.488727 24854 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 02:15:43.490710 24854 catalog_manager.cc:1504] Loading token signing keys...
I20260928 02:15:43.502071 24854 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5cb923917cdd4dd8bd6ebdd8b7634d16: Generated new TSK 0
I20260928 02:15:43.503304 24854 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 02:15:43.516492 7085 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTlaUuc/build/asan/bin/kudu
/tmp/dist-test-taskTlaUuc/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790561432251112-7085-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790561432251112-7085-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790561432251112-7085-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790561432251112-7085-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.235.65:0
--local_ip_for_outbound_sockets=127.6.235.65
--webserver_interface=127.6.235.65
--webserver_port=0
--tserver_master_addrs=127.6.235.126:38927
--builtin_ntp_servers=127.6.235.84:46507
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260928 02:15:43.872220 24866 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:15:43.872545 24866 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:15:43.872704 24866 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:15:43.881610 24866 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:15:43.881745 24866 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:15:43.881947 24866 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.235.65
I20260928 02:15:43.892746 24866 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.235.84:46507
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790561432251112-7085-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790561432251112-7085-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.235.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790561432251112-7085-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.6.235.65
--webserver_port=0
--tserver_master_addrs=127.6.235.126:38927
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.235.65
--log_dir=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790561432251112-7085-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 00:51:59 UTC on ccycloud.quasar-axgkwn.root.comops.site
build id 4676464
ASAN enabled
I20260928 02:15:43.894241 24866 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:15:43.896432 24866 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:15:43.912026 24872 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:15:43.912153 24874 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:15:43.912442 24871 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:15:43.913322 24866 server_base.cc:1034] running on GCE node
I20260928 02:15:43.914039 24866 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:15:43.915206 24866 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:15:43.916462 24866 hybrid_clock.cc:648] HybridClock initialized: now 1790561743916394 us; error 62 us; skew 500 ppm
I20260928 02:15:43.916877 24866 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260928 02:15:43.919629 24866 webserver.cc:460] Webserver started at http://127.6.235.65:35553/ using document root <none> and password file <none>
I20260928 02:15:43.920445 24866 fs_manager.cc:359] Metadata directory not provided
I20260928 02:15:43.920593 24866 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:15:43.921012 24866 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:15:43.924105 24866 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790561432251112-7085-0/minicluster-data/ts-0/data/instance:
uuid: "ac4678909a0b419daf9990f634f7ff5e"
format_stamp: "Formatted at 2026-09-28 02:15:43 on dist-test-slave-86gv"
server_key: "21866eec9f968abbc63d226a285e3fb4"
server_key_iv: "eceb2d05b35ef09e0a5458f93c18a0e4"
server_key_version: "encryptionkey@0"
I20260928 02:15:43.925071 24866 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790561432251112-7085-0/minicluster-data/ts-0/wal/instance:
uuid: "ac4678909a0b419daf9990f634f7ff5e"
format_stamp: "Formatted at 2026-09-28 02:15:43 on dist-test-slave-86gv"
server_key: "21866eec9f968abbc63d226a285e3fb4"
server_key_iv: "eceb2d05b35ef09e0a5458f93c18a0e4"
server_key_version: "encryptionkey@0"
I20260928 02:15:43.930872 24866 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260928 02:15:43.934932 24880 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:15:43.936657 24866 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260928 02:15:43.936964 24866 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790561432251112-7085-0/minicluster-data/ts-0/data,/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790561432251112-7085-0/minicluster-data/ts-0/wal
uuid: "ac4678909a0b419daf9990f634f7ff5e"
format_stamp: "Formatted at 2026-09-28 02:15:43 on dist-test-slave-86gv"
server_key: "21866eec9f968abbc63d226a285e3fb4"
server_key_iv: "eceb2d05b35ef09e0a5458f93c18a0e4"
server_key_version: "encryptionkey@0"
I20260928 02:15:43.937218 24866 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790561432251112-7085-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790561432251112-7085-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790561432251112-7085-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:15:43.951531 24866 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:15:43.952455 24866 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:15:43.952796 24866 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:15:43.953893 24866 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:15:43.955480 24866 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:15:43.955644 24866 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:15:43.955783 24866 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:15:43.955901 24866 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.001s sys 0.000s
I20260928 02:15:43.990109 24866 rpc_server.cc:307] RPC server started. Bound to: 127.6.235.65:34425
I20260928 02:15:43.990199 24992 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.235.65:34425 every 8 connection(s)
I20260928 02:15:43.991582 24866 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790561432251112-7085-0/minicluster-data/ts-0/data/info.pb
I20260928 02:15:43.996798 7085 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTlaUuc/build/asan/bin/kudu as pid 24866
I20260928 02:15:43.997023 7085 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790561432251112-7085-0/minicluster-data/ts-0/wal/instance
I20260928 02:15:43.997457 7085 external_mini_cluster.cc:1428] Setting key 0bac44c6b5bca091ec1708400274159e
I20260928 02:15:44.000234 7085 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTlaUuc/build/asan/bin/kudu
/tmp/dist-test-taskTlaUuc/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790561432251112-7085-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790561432251112-7085-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790561432251112-7085-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790561432251112-7085-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.235.66:0
--local_ip_for_outbound_sockets=127.6.235.66
--webserver_interface=127.6.235.66
--webserver_port=0
--tserver_master_addrs=127.6.235.126:38927
--builtin_ntp_servers=127.6.235.84:46507
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260928 02:15:44.004868 24995 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:15:43.994284 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.235.65:38089 (local address 127.6.235.126:38927)
0928 02:15:43.994736 (+ 452us) server_negotiation.cc:206] Beginning negotiation
0928 02:15:43.994741 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 02:15:43.995910 (+ 1169us) server_negotiation.cc:407] Connection header received
0928 02:15:43.997914 (+ 2004us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:15:43.997920 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:15:43.998015 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:15:43.998155 (+ 140us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:15:43.999279 (+ 1124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:15:44.000045 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:15:44.001810 (+ 1765us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:15:44.002384 (+ 574us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:15:44.002524 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:15:44.004014 (+ 1490us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:15:44.004019 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:15:44.004022 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 02:15:44.004043 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:15:44.004082 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 02:15:44.004094 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:15:44.004202 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0928 02:15:44.004428 (+ 226us) server_negotiation.cc:299] Negotiation successful
0928 02:15:44.004532 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":339,"thread_start_us":236,"threads_started":1}
I20260928 02:15:44.006245 24996 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:15:43.994845 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.235.126:38927 (local address 127.6.235.65:38089)
0928 02:15:43.995746 (+ 901us) negotiation.cc:107] Waiting for socket to connect
0928 02:15:43.995790 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0928 02:15:43.997634 (+ 1844us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:15:43.998390 (+ 756us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:15:43.998399 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:15:43.998508 (+ 109us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:15:43.999109 (+ 601us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:15:43.999125 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:15:44.000219 (+ 1094us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:15:44.000223 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:15:44.001655 (+ 1432us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:15:44.001667 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:15:44.002613 (+ 946us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:15:44.002620 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:15:44.002874 (+ 254us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:15:44.003625 (+ 751us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 02:15:44.003628 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:15:44.003715 (+ 87us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 02:15:44.003801 (+ 86us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:15:44.004200 (+ 399us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:15:44.004212 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:15:44.004223 (+ 11us) client_negotiation.cc:769] Sending connection context
0928 02:15:44.004419 (+ 196us) client_negotiation.cc:240] Negotiation successful
0928 02:15:44.004872 (+ 453us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":451,"thread_start_us":276,"threads_started":1}
I20260928 02:15:44.007907 24993 heartbeater.cc:347] Connected to a master server at 127.6.235.126:38927
I20260928 02:15:44.008253 24993 heartbeater.cc:464] Registering TS with master...
I20260928 02:15:44.009220 24993 heartbeater.cc:511] Master 127.6.235.126:38927 requested a full tablet report, sending...
I20260928 02:15:44.011497 24807 ts_manager.cc:194] Registered new tserver with Master: ac4678909a0b419daf9990f634f7ff5e (127.6.235.65:34425)
I20260928 02:15:44.013177 24807 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.6.235.65:38089
W20260928 02:15:44.353849 24997 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:15:44.354260 24997 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:15:44.354422 24997 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:15:44.362537 24997 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:15:44.362705 24997 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:15:44.362890 24997 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.235.66
I20260928 02:15:44.373032 24997 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.235.84:46507
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790561432251112-7085-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790561432251112-7085-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.235.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790561432251112-7085-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.6.235.66
--webserver_port=0
--tserver_master_addrs=127.6.235.126:38927
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.235.66
--log_dir=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790561432251112-7085-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 00:51:59 UTC on ccycloud.quasar-axgkwn.root.comops.site
build id 4676464
ASAN enabled
I20260928 02:15:44.374544 24997 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:15:44.376609 24997 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:15:44.391796 25005 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:15:44.391839 25002 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:15:44.391880 25003 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:15:44.392957 24997 server_base.cc:1034] running on GCE node
I20260928 02:15:44.393702 24997 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:15:44.394860 24997 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:15:44.396106 24997 hybrid_clock.cc:648] HybridClock initialized: now 1790561744396034 us; error 70 us; skew 500 ppm
I20260928 02:15:44.396423 24997 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260928 02:15:44.398900 24997 webserver.cc:460] Webserver started at http://127.6.235.66:37747/ using document root <none> and password file <none>
I20260928 02:15:44.399612 24997 fs_manager.cc:359] Metadata directory not provided
I20260928 02:15:44.399713 24997 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:15:44.399983 24997 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:15:44.402979 24997 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790561432251112-7085-0/minicluster-data/ts-1/data/instance:
uuid: "f5aa645bccb54161aa1c7645b66a917d"
format_stamp: "Formatted at 2026-09-28 02:15:44 on dist-test-slave-86gv"
server_key: "9200a228bca7d706ed332bc305a40137"
server_key_iv: "67d585a922f2d77e0b9b8c220422e74f"
server_key_version: "encryptionkey@0"
I20260928 02:15:44.403735 24997 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790561432251112-7085-0/minicluster-data/ts-1/wal/instance:
uuid: "f5aa645bccb54161aa1c7645b66a917d"
format_stamp: "Formatted at 2026-09-28 02:15:44 on dist-test-slave-86gv"
server_key: "9200a228bca7d706ed332bc305a40137"
server_key_iv: "67d585a922f2d77e0b9b8c220422e74f"
server_key_version: "encryptionkey@0"
I20260928 02:15:44.409284 24997 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.003s
I20260928 02:15:44.413355 25011 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:15:44.414937 24997 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260928 02:15:44.415118 24997 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790561432251112-7085-0/minicluster-data/ts-1/data,/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790561432251112-7085-0/minicluster-data/ts-1/wal
uuid: "f5aa645bccb54161aa1c7645b66a917d"
format_stamp: "Formatted at 2026-09-28 02:15:44 on dist-test-slave-86gv"
server_key: "9200a228bca7d706ed332bc305a40137"
server_key_iv: "67d585a922f2d77e0b9b8c220422e74f"
server_key_version: "encryptionkey@0"
I20260928 02:15:44.415359 24997 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790561432251112-7085-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790561432251112-7085-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790561432251112-7085-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:15:44.442858 24997 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:15:44.443774 24997 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:15:44.444144 24997 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:15:44.445729 24997 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:15:44.447311 24997 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:15:44.447445 24997 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:15:44.447587 24997 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:15:44.447690 24997 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:15:44.482153 24997 rpc_server.cc:307] RPC server started. Bound to: 127.6.235.66:40613
I20260928 02:15:44.482249 25123 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.235.66:40613 every 8 connection(s)
I20260928 02:15:44.483565 24997 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790561432251112-7085-0/minicluster-data/ts-1/data/info.pb
I20260928 02:15:44.488848 7085 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTlaUuc/build/asan/bin/kudu as pid 24997
I20260928 02:15:44.489221 7085 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790561432251112-7085-0/minicluster-data/ts-1/wal/instance
I20260928 02:15:44.489719 7085 external_mini_cluster.cc:1428] Setting key b82a8802968dfd2cc71901e92f8e2b1d
I20260928 02:15:44.496522 7085 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTlaUuc/build/asan/bin/kudu
/tmp/dist-test-taskTlaUuc/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790561432251112-7085-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790561432251112-7085-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790561432251112-7085-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790561432251112-7085-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.235.67:0
--local_ip_for_outbound_sockets=127.6.235.67
--webserver_interface=127.6.235.67
--webserver_port=0
--tserver_master_addrs=127.6.235.126:38927
--builtin_ntp_servers=127.6.235.84:46507
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260928 02:15:44.496841 24995 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:15:44.486456 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.235.66:60661 (local address 127.6.235.126:38927)
0928 02:15:44.486656 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0928 02:15:44.486661 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 02:15:44.488122 (+ 1461us) server_negotiation.cc:407] Connection header received
0928 02:15:44.490190 (+ 2068us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:15:44.490196 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:15:44.490295 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:15:44.490410 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:15:44.491555 (+ 1145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:15:44.492291 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:15:44.493957 (+ 1666us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:15:44.494368 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:15:44.494517 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:15:44.495988 (+ 1471us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:15:44.495993 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:15:44.495996 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 02:15:44.496014 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:15:44.496046 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 02:15:44.496059 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:15:44.496154 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0928 02:15:44.496407 (+ 253us) server_negotiation.cc:299] Negotiation successful
0928 02:15:44.496506 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260928 02:15:44.497952 25126 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:15:44.486991 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.235.126:38927 (local address 127.6.235.66:60661)
0928 02:15:44.487951 (+ 960us) negotiation.cc:107] Waiting for socket to connect
0928 02:15:44.487990 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0928 02:15:44.489891 (+ 1901us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:15:44.490659 (+ 768us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:15:44.490667 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:15:44.490779 (+ 112us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:15:44.491378 (+ 599us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:15:44.491399 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:15:44.492447 (+ 1048us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:15:44.492460 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:15:44.493804 (+ 1344us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:15:44.493816 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:15:44.494611 (+ 795us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:15:44.494618 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:15:44.494860 (+ 242us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:15:44.495658 (+ 798us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 02:15:44.495661 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:15:44.495707 (+ 46us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 02:15:44.495787 (+ 80us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:15:44.496175 (+ 388us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:15:44.496193 (+ 18us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:15:44.496200 (+ 7us) client_negotiation.cc:769] Sending connection context
0928 02:15:44.496385 (+ 185us) client_negotiation.cc:240] Negotiation successful
0928 02:15:44.496809 (+ 424us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":481,"thread_start_us":296,"threads_started":1}
I20260928 02:15:44.499483 25124 heartbeater.cc:347] Connected to a master server at 127.6.235.126:38927
I20260928 02:15:44.499886 25124 heartbeater.cc:464] Registering TS with master...
I20260928 02:15:44.500821 25124 heartbeater.cc:511] Master 127.6.235.126:38927 requested a full tablet report, sending...
I20260928 02:15:44.502671 24807 ts_manager.cc:194] Registered new tserver with Master: f5aa645bccb54161aa1c7645b66a917d (127.6.235.66:40613)
I20260928 02:15:44.503907 24807 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.6.235.66:60661
W20260928 02:15:44.846050 25127 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:15:44.846346 25127 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:15:44.846499 25127 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:15:44.854255 25127 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:15:44.854405 25127 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:15:44.854581 25127 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.235.67
I20260928 02:15:44.864154 25127 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.235.84:46507
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790561432251112-7085-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790561432251112-7085-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.235.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790561432251112-7085-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.6.235.67
--webserver_port=0
--tserver_master_addrs=127.6.235.126:38927
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.235.67
--log_dir=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790561432251112-7085-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 00:51:59 UTC on ccycloud.quasar-axgkwn.root.comops.site
build id 4676464
ASAN enabled
I20260928 02:15:44.865628 25127 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:15:44.867533 25127 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:15:44.881639 25135 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:15:44.881651 25133 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:15:44.882493 25132 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:15:44.882475 25127 server_base.cc:1034] running on GCE node
I20260928 02:15:44.883265 25127 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:15:44.884280 25127 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:15:44.885485 25127 hybrid_clock.cc:648] HybridClock initialized: now 1790561744885412 us; error 63 us; skew 500 ppm
I20260928 02:15:44.885825 25127 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260928 02:15:44.888239 25127 webserver.cc:460] Webserver started at http://127.6.235.67:35873/ using document root <none> and password file <none>
I20260928 02:15:44.889055 25127 fs_manager.cc:359] Metadata directory not provided
I20260928 02:15:44.889187 25127 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:15:44.889539 25127 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:15:44.892611 25127 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790561432251112-7085-0/minicluster-data/ts-2/data/instance:
uuid: "98febb23b62c463d9f1770c3c07481f3"
format_stamp: "Formatted at 2026-09-28 02:15:44 on dist-test-slave-86gv"
server_key: "016f0c2178f360cde9c698eff87a6803"
server_key_iv: "22897c02378a1d899e854b2e30404b10"
server_key_version: "encryptionkey@0"
I20260928 02:15:44.893481 25127 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790561432251112-7085-0/minicluster-data/ts-2/wal/instance:
uuid: "98febb23b62c463d9f1770c3c07481f3"
format_stamp: "Formatted at 2026-09-28 02:15:44 on dist-test-slave-86gv"
server_key: "016f0c2178f360cde9c698eff87a6803"
server_key_iv: "22897c02378a1d899e854b2e30404b10"
server_key_version: "encryptionkey@0"
I20260928 02:15:44.899055 25127 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.004s
I20260928 02:15:44.902712 25141 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:15:44.904098 25127 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260928 02:15:44.904311 25127 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790561432251112-7085-0/minicluster-data/ts-2/data,/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790561432251112-7085-0/minicluster-data/ts-2/wal
uuid: "98febb23b62c463d9f1770c3c07481f3"
format_stamp: "Formatted at 2026-09-28 02:15:44 on dist-test-slave-86gv"
server_key: "016f0c2178f360cde9c698eff87a6803"
server_key_iv: "22897c02378a1d899e854b2e30404b10"
server_key_version: "encryptionkey@0"
I20260928 02:15:44.904537 25127 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790561432251112-7085-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790561432251112-7085-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790561432251112-7085-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:15:44.917824 25127 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:15:44.918773 25127 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:15:44.919178 25127 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:15:44.920272 25127 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:15:44.921964 25127 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:15:44.922056 25127 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:15:44.922142 25127 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:15:44.922200 25127 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:15:44.957233 25127 rpc_server.cc:307] RPC server started. Bound to: 127.6.235.67:39559
I20260928 02:15:44.957255 25253 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.235.67:39559 every 8 connection(s)
I20260928 02:15:44.958639 25127 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790561432251112-7085-0/minicluster-data/ts-2/data/info.pb
I20260928 02:15:44.967110 7085 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTlaUuc/build/asan/bin/kudu as pid 25127
I20260928 02:15:44.967305 7085 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790561432251112-7085-0/minicluster-data/ts-2/wal/instance
I20260928 02:15:44.967847 7085 external_mini_cluster.cc:1428] Setting key 2b45260b52d94ae7c3ecb2c5d2504229
I20260928 02:15:44.972750 24995 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:15:44.961442 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.235.67:41577 (local address 127.6.235.126:38927)
0928 02:15:44.961699 (+ 257us) server_negotiation.cc:206] Beginning negotiation
0928 02:15:44.961706 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 02:15:44.963437 (+ 1731us) server_negotiation.cc:407] Connection header received
0928 02:15:44.965284 (+ 1847us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:15:44.965288 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:15:44.965358 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:15:44.965451 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:15:44.966676 (+ 1225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:15:44.967808 (+ 1132us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:15:44.969900 (+ 2092us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:15:44.970363 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:15:44.970513 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:15:44.971883 (+ 1370us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:15:44.971888 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:15:44.971891 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 02:15:44.971908 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:15:44.971939 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 02:15:44.971951 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:15:44.972043 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0928 02:15:44.972316 (+ 273us) server_negotiation.cc:299] Negotiation successful
0928 02:15:44.972408 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260928 02:15:44.973788 25256 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:15:44.962165 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.235.126:38927 (local address 127.6.235.67:41577)
0928 02:15:44.963258 (+ 1093us) negotiation.cc:107] Waiting for socket to connect
0928 02:15:44.963305 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0928 02:15:44.965014 (+ 1709us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:15:44.965693 (+ 679us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:15:44.965702 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:15:44.965794 (+ 92us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:15:44.966422 (+ 628us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:15:44.966438 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:15:44.968010 (+ 1572us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:15:44.968014 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:15:44.969706 (+ 1692us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:15:44.969718 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:15:44.970618 (+ 900us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:15:44.970625 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:15:44.970898 (+ 273us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:15:44.971561 (+ 663us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 02:15:44.971564 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:15:44.971603 (+ 39us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 02:15:44.971685 (+ 82us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:15:44.972058 (+ 373us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:15:44.972071 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:15:44.972086 (+ 15us) client_negotiation.cc:769] Sending connection context
0928 02:15:44.972303 (+ 217us) client_negotiation.cc:240] Negotiation successful
0928 02:15:44.972709 (+ 406us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":572,"thread_start_us":365,"threads_started":1}
I20260928 02:15:44.975175 25254 heartbeater.cc:347] Connected to a master server at 127.6.235.126:38927
I20260928 02:15:44.975490 25254 heartbeater.cc:464] Registering TS with master...
I20260928 02:15:44.976281 25254 heartbeater.cc:511] Master 127.6.235.126:38927 requested a full tablet report, sending...
I20260928 02:15:44.977756 24807 ts_manager.cc:194] Registered new tserver with Master: 98febb23b62c463d9f1770c3c07481f3 (127.6.235.67:39559)
I20260928 02:15:44.978564 24807 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.6.235.67:41577
I20260928 02:15:44.983224 7085 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 02:15:44.994287 24995 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:15:44.987291 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45516 (local address 127.6.235.126:38927)
0928 02:15:44.987505 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0928 02:15:44.987515 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 02:15:44.987964 (+ 449us) server_negotiation.cc:407] Connection header received
0928 02:15:44.988101 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:15:44.988105 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:15:44.988176 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:15:44.988260 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:15:44.988740 (+ 480us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:15:44.989621 (+ 881us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:15:44.991530 (+ 1909us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:15:44.992181 (+ 651us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:15:44.992314 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:15:44.993527 (+ 1213us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:15:44.993533 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:15:44.993538 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 02:15:44.993562 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:15:44.993602 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 02:15:44.993618 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:15:44.993717 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0928 02:15:44.993879 (+ 162us) server_negotiation.cc:299] Negotiation successful
0928 02:15:44.993974 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260928 02:15:44.998531 24807 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45516:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 02:15:45.005594 24807 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 02:15:45.016561 24993 heartbeater.cc:503] Master 127.6.235.126:38927 was elected leader, sending a full tablet report...
I20260928 02:15:45.033987 25266 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:15:45.019443 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35510 (local address 127.6.235.65:34425)
0928 02:15:45.020089 (+ 646us) server_negotiation.cc:206] Beginning negotiation
0928 02:15:45.020095 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 02:15:45.021114 (+ 1019us) server_negotiation.cc:407] Connection header received
0928 02:15:45.021580 (+ 466us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:15:45.021591 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:15:45.021948 (+ 357us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:15:45.022113 (+ 165us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:15:45.023294 (+ 1181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:15:45.025493 (+ 2199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:15:45.029351 (+ 3858us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:15:45.031550 (+ 2199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:15:45.031715 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:15:45.032122 (+ 407us) server_negotiation.cc:1035] Waiting for connection context
0928 02:15:45.033130 (+ 1008us) server_negotiation.cc:299] Negotiation successful
0928 02:15:45.033409 (+ 279us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":458,"thread_start_us":1068,"threads_started":1}
I20260928 02:15:45.035598 25268 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:15:45.019483 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.235.65:34425 (local address 127.0.0.1:35510)
0928 02:15:45.020916 (+ 1433us) negotiation.cc:107] Waiting for socket to connect
0928 02:15:45.020979 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0928 02:15:45.021302 (+ 323us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:15:45.022496 (+ 1194us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:15:45.022507 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:15:45.022550 (+ 43us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:15:45.023061 (+ 511us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:15:45.023077 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:15:45.025769 (+ 2692us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:15:45.025778 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:15:45.029032 (+ 3254us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:15:45.029058 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:15:45.032373 (+ 3315us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:15:45.032382 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:15:45.032736 (+ 354us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:15:45.032812 (+ 76us) client_negotiation.cc:769] Sending connection context
0928 02:15:45.034700 (+ 1888us) client_negotiation.cc:240] Negotiation successful
0928 02:15:45.035048 (+ 348us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1052,"thread_start_us":14978,"threads_started":1}
I20260928 02:15:45.040721 24928 tablet_service.cc:1467] Processing CreateTablet for tablet 3c53e1d19c0e48b29c04acdd3fe8fd41 (DEFAULT_TABLE table=test-table [id=718138cca35742a29a2e1fcfa2824b7f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:15:45.042455 24928 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3c53e1d19c0e48b29c04acdd3fe8fd41. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:15:45.043522 25270 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:15:45.023073 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.235.66:40613 (local address 127.0.0.1:34196)
0928 02:15:45.035671 (+ 12598us) negotiation.cc:107] Waiting for socket to connect
0928 02:15:45.035694 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0928 02:15:45.035825 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:15:45.036741 (+ 916us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:15:45.036745 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:15:45.036771 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:15:45.037145 (+ 374us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:15:45.037159 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:15:45.038914 (+ 1755us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:15:45.038920 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:15:45.040876 (+ 1956us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:15:45.040936 (+ 60us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:15:45.042558 (+ 1622us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:15:45.042565 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:15:45.042805 (+ 240us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:15:45.042845 (+ 40us) client_negotiation.cc:769] Sending connection context
0928 02:15:45.042963 (+ 118us) client_negotiation.cc:240] Negotiation successful
0928 02:15:45.043126 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12465,"mutex_wait_us":4,"thread_start_us":12315,"threads_started":1}
I20260928 02:15:45.043627 25271 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:15:45.023693 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34196 (local address 127.6.235.66:40613)
0928 02:15:45.035182 (+ 11489us) server_negotiation.cc:206] Beginning negotiation
0928 02:15:45.035196 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0928 02:15:45.035959 (+ 763us) server_negotiation.cc:407] Connection header received
0928 02:15:45.036112 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:15:45.036122 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:15:45.036415 (+ 293us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:15:45.036644 (+ 229us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:15:45.037385 (+ 741us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:15:45.038787 (+ 1402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:15:45.041127 (+ 2340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:15:45.042355 (+ 1228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:15:45.042518 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:15:45.042778 (+ 260us) server_negotiation.cc:1035] Waiting for connection context
0928 02:15:45.043091 (+ 313us) server_negotiation.cc:299] Negotiation successful
0928 02:15:45.043268 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11299,"thread_start_us":11154,"threads_started":1}
I20260928 02:15:45.045912 25267 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:15:45.020097 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.235.67:39559 (local address 127.0.0.1:49688)
0928 02:15:45.034169 (+ 14072us) negotiation.cc:107] Waiting for socket to connect
0928 02:15:45.034196 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0928 02:15:45.034363 (+ 167us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:15:45.038064 (+ 3701us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:15:45.038069 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:15:45.038097 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:15:45.038437 (+ 340us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:15:45.038449 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:15:45.040329 (+ 1880us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:15:45.040334 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:15:45.042772 (+ 2438us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:15:45.042790 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:15:45.045077 (+ 2287us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:15:45.045084 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:15:45.045280 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:15:45.045332 (+ 52us) client_negotiation.cc:769] Sending connection context
0928 02:15:45.045436 (+ 104us) client_negotiation.cc:240] Negotiation successful
0928 02:15:45.045571 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13836,"mutex_wait_us":2,"thread_start_us":615,"threads_started":1}
I20260928 02:15:45.045992 25269 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:15:45.020665 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49688 (local address 127.6.235.67:39559)
0928 02:15:45.037422 (+ 16757us) server_negotiation.cc:206] Beginning negotiation
0928 02:15:45.037431 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 02:15:45.037466 (+ 35us) server_negotiation.cc:407] Connection header received
0928 02:15:45.037589 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:15:45.037597 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:15:45.037852 (+ 255us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:15:45.038055 (+ 203us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:15:45.038634 (+ 579us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:15:45.040162 (+ 1528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:15:45.042989 (+ 2827us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:15:45.044858 (+ 1869us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:15:45.045030 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:15:45.045312 (+ 282us) server_negotiation.cc:1035] Waiting for connection context
0928 02:15:45.045491 (+ 179us) server_negotiation.cc:299] Negotiation successful
0928 02:15:45.045660 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16563,"thread_start_us":16405,"threads_started":1}
I20260928 02:15:45.047801 25059 tablet_service.cc:1467] Processing CreateTablet for tablet 3c53e1d19c0e48b29c04acdd3fe8fd41 (DEFAULT_TABLE table=test-table [id=718138cca35742a29a2e1fcfa2824b7f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:15:45.049741 25059 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3c53e1d19c0e48b29c04acdd3fe8fd41. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:15:45.050549 25189 tablet_service.cc:1467] Processing CreateTablet for tablet 3c53e1d19c0e48b29c04acdd3fe8fd41 (DEFAULT_TABLE table=test-table [id=718138cca35742a29a2e1fcfa2824b7f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:15:45.052114 25189 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3c53e1d19c0e48b29c04acdd3fe8fd41. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:15:45.057102 25272 tablet_bootstrap.cc:492] T 3c53e1d19c0e48b29c04acdd3fe8fd41 P ac4678909a0b419daf9990f634f7ff5e: Bootstrap starting.
I20260928 02:15:45.060415 25272 tablet_bootstrap.cc:654] T 3c53e1d19c0e48b29c04acdd3fe8fd41 P ac4678909a0b419daf9990f634f7ff5e: Neither blocks nor log segments found. Creating new log.
I20260928 02:15:45.061708 25272 log.cc:824] T 3c53e1d19c0e48b29c04acdd3fe8fd41 P ac4678909a0b419daf9990f634f7ff5e: Log is configured to *not* fsync() on all Append() calls
I20260928 02:15:45.065984 25274 tablet_bootstrap.cc:492] T 3c53e1d19c0e48b29c04acdd3fe8fd41 P f5aa645bccb54161aa1c7645b66a917d: Bootstrap starting.
I20260928 02:15:45.067909 25272 tablet_bootstrap.cc:492] T 3c53e1d19c0e48b29c04acdd3fe8fd41 P ac4678909a0b419daf9990f634f7ff5e: No bootstrap required, opened a new log
I20260928 02:15:45.068236 25272 ts_tablet_manager.cc:1397] T 3c53e1d19c0e48b29c04acdd3fe8fd41 P ac4678909a0b419daf9990f634f7ff5e: Time spent bootstrapping tablet: real 0.011s user 0.003s sys 0.006s
I20260928 02:15:45.069706 25275 tablet_bootstrap.cc:492] T 3c53e1d19c0e48b29c04acdd3fe8fd41 P 98febb23b62c463d9f1770c3c07481f3: Bootstrap starting.
I20260928 02:15:45.070788 25274 tablet_bootstrap.cc:654] T 3c53e1d19c0e48b29c04acdd3fe8fd41 P f5aa645bccb54161aa1c7645b66a917d: Neither blocks nor log segments found. Creating new log.
I20260928 02:15:45.072274 25274 log.cc:824] T 3c53e1d19c0e48b29c04acdd3fe8fd41 P f5aa645bccb54161aa1c7645b66a917d: Log is configured to *not* fsync() on all Append() calls
I20260928 02:15:45.073426 25275 tablet_bootstrap.cc:654] T 3c53e1d19c0e48b29c04acdd3fe8fd41 P 98febb23b62c463d9f1770c3c07481f3: Neither blocks nor log segments found. Creating new log.
I20260928 02:15:45.074745 25275 log.cc:824] T 3c53e1d19c0e48b29c04acdd3fe8fd41 P 98febb23b62c463d9f1770c3c07481f3: Log is configured to *not* fsync() on all Append() calls
I20260928 02:15:45.076207 25274 tablet_bootstrap.cc:492] T 3c53e1d19c0e48b29c04acdd3fe8fd41 P f5aa645bccb54161aa1c7645b66a917d: No bootstrap required, opened a new log
I20260928 02:15:45.076502 25274 ts_tablet_manager.cc:1397] T 3c53e1d19c0e48b29c04acdd3fe8fd41 P f5aa645bccb54161aa1c7645b66a917d: Time spent bootstrapping tablet: real 0.011s user 0.006s sys 0.001s
I20260928 02:15:45.079536 25275 tablet_bootstrap.cc:492] T 3c53e1d19c0e48b29c04acdd3fe8fd41 P 98febb23b62c463d9f1770c3c07481f3: No bootstrap required, opened a new log
I20260928 02:15:45.079156 25272 raft_consensus.cc:348] T 3c53e1d19c0e48b29c04acdd3fe8fd41 P ac4678909a0b419daf9990f634f7ff5e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac4678909a0b419daf9990f634f7ff5e" member_type: VOTER last_known_addr { host: "127.6.235.65" port: 34425 } } peers { permanent_uuid: "98febb23b62c463d9f1770c3c07481f3" member_type: VOTER last_known_addr { host: "127.6.235.67" port: 39559 } } peers { permanent_uuid: "f5aa645bccb54161aa1c7645b66a917d" member_type: VOTER last_known_addr { host: "127.6.235.66" port: 40613 } }
I20260928 02:15:45.079692 25272 raft_consensus.cc:374] T 3c53e1d19c0e48b29c04acdd3fe8fd41 P ac4678909a0b419daf9990f634f7ff5e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:15:45.079856 25272 raft_consensus.cc:729] T 3c53e1d19c0e48b29c04acdd3fe8fd41 P ac4678909a0b419daf9990f634f7ff5e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ac4678909a0b419daf9990f634f7ff5e, State: Initialized, Role: FOLLOWER
I20260928 02:15:45.079875 25275 ts_tablet_manager.cc:1397] T 3c53e1d19c0e48b29c04acdd3fe8fd41 P 98febb23b62c463d9f1770c3c07481f3: Time spent bootstrapping tablet: real 0.010s user 0.008s sys 0.000s
I20260928 02:15:45.080569 25272 consensus_queue.cc:260] T 3c53e1d19c0e48b29c04acdd3fe8fd41 P ac4678909a0b419daf9990f634f7ff5e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac4678909a0b419daf9990f634f7ff5e" member_type: VOTER last_known_addr { host: "127.6.235.65" port: 34425 } } peers { permanent_uuid: "98febb23b62c463d9f1770c3c07481f3" member_type: VOTER last_known_addr { host: "127.6.235.67" port: 39559 } } peers { permanent_uuid: "f5aa645bccb54161aa1c7645b66a917d" member_type: VOTER last_known_addr { host: "127.6.235.66" port: 40613 } }
I20260928 02:15:45.082410 25272 ts_tablet_manager.cc:1428] T 3c53e1d19c0e48b29c04acdd3fe8fd41 P ac4678909a0b419daf9990f634f7ff5e: Time spent starting tablet: real 0.014s user 0.011s sys 0.004s
I20260928 02:15:45.086623 25274 raft_consensus.cc:348] T 3c53e1d19c0e48b29c04acdd3fe8fd41 P f5aa645bccb54161aa1c7645b66a917d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac4678909a0b419daf9990f634f7ff5e" member_type: VOTER last_known_addr { host: "127.6.235.65" port: 34425 } } peers { permanent_uuid: "98febb23b62c463d9f1770c3c07481f3" member_type: VOTER last_known_addr { host: "127.6.235.67" port: 39559 } } peers { permanent_uuid: "f5aa645bccb54161aa1c7645b66a917d" member_type: VOTER last_known_addr { host: "127.6.235.66" port: 40613 } }
I20260928 02:15:45.087098 25274 raft_consensus.cc:374] T 3c53e1d19c0e48b29c04acdd3fe8fd41 P f5aa645bccb54161aa1c7645b66a917d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:15:45.087247 25274 raft_consensus.cc:729] T 3c53e1d19c0e48b29c04acdd3fe8fd41 P f5aa645bccb54161aa1c7645b66a917d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f5aa645bccb54161aa1c7645b66a917d, State: Initialized, Role: FOLLOWER
I20260928 02:15:45.087955 25274 consensus_queue.cc:260] T 3c53e1d19c0e48b29c04acdd3fe8fd41 P f5aa645bccb54161aa1c7645b66a917d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac4678909a0b419daf9990f634f7ff5e" member_type: VOTER last_known_addr { host: "127.6.235.65" port: 34425 } } peers { permanent_uuid: "98febb23b62c463d9f1770c3c07481f3" member_type: VOTER last_known_addr { host: "127.6.235.67" port: 39559 } } peers { permanent_uuid: "f5aa645bccb54161aa1c7645b66a917d" member_type: VOTER last_known_addr { host: "127.6.235.66" port: 40613 } }
I20260928 02:15:45.089324 25124 heartbeater.cc:503] Master 127.6.235.126:38927 was elected leader, sending a full tablet report...
I20260928 02:15:45.089993 25274 ts_tablet_manager.cc:1428] T 3c53e1d19c0e48b29c04acdd3fe8fd41 P f5aa645bccb54161aa1c7645b66a917d: Time spent starting tablet: real 0.013s user 0.007s sys 0.008s
I20260928 02:15:45.090024 25275 raft_consensus.cc:348] T 3c53e1d19c0e48b29c04acdd3fe8fd41 P 98febb23b62c463d9f1770c3c07481f3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac4678909a0b419daf9990f634f7ff5e" member_type: VOTER last_known_addr { host: "127.6.235.65" port: 34425 } } peers { permanent_uuid: "98febb23b62c463d9f1770c3c07481f3" member_type: VOTER last_known_addr { host: "127.6.235.67" port: 39559 } } peers { permanent_uuid: "f5aa645bccb54161aa1c7645b66a917d" member_type: VOTER last_known_addr { host: "127.6.235.66" port: 40613 } }
I20260928 02:15:45.090550 25275 raft_consensus.cc:374] T 3c53e1d19c0e48b29c04acdd3fe8fd41 P 98febb23b62c463d9f1770c3c07481f3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:15:45.090680 25275 raft_consensus.cc:729] T 3c53e1d19c0e48b29c04acdd3fe8fd41 P 98febb23b62c463d9f1770c3c07481f3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 98febb23b62c463d9f1770c3c07481f3, State: Initialized, Role: FOLLOWER
I20260928 02:15:45.091336 25275 consensus_queue.cc:260] T 3c53e1d19c0e48b29c04acdd3fe8fd41 P 98febb23b62c463d9f1770c3c07481f3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac4678909a0b419daf9990f634f7ff5e" member_type: VOTER last_known_addr { host: "127.6.235.65" port: 34425 } } peers { permanent_uuid: "98febb23b62c463d9f1770c3c07481f3" member_type: VOTER last_known_addr { host: "127.6.235.67" port: 39559 } } peers { permanent_uuid: "f5aa645bccb54161aa1c7645b66a917d" member_type: VOTER last_known_addr { host: "127.6.235.66" port: 40613 } }
I20260928 02:15:45.092439 25254 heartbeater.cc:503] Master 127.6.235.126:38927 was elected leader, sending a full tablet report...
I20260928 02:15:45.093123 25275 ts_tablet_manager.cc:1428] T 3c53e1d19c0e48b29c04acdd3fe8fd41 P 98febb23b62c463d9f1770c3c07481f3: Time spent starting tablet: real 0.013s user 0.011s sys 0.004s
I20260928 02:15:45.118970 25280 raft_consensus.cc:482] T 3c53e1d19c0e48b29c04acdd3fe8fd41 P 98febb23b62c463d9f1770c3c07481f3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 02:15:45.119356 25280 raft_consensus.cc:504] T 3c53e1d19c0e48b29c04acdd3fe8fd41 P 98febb23b62c463d9f1770c3c07481f3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac4678909a0b419daf9990f634f7ff5e" member_type: VOTER last_known_addr { host: "127.6.235.65" port: 34425 } } peers { permanent_uuid: "98febb23b62c463d9f1770c3c07481f3" member_type: VOTER last_known_addr { host: "127.6.235.67" port: 39559 } } peers { permanent_uuid: "f5aa645bccb54161aa1c7645b66a917d" member_type: VOTER last_known_addr { host: "127.6.235.66" port: 40613 } }
I20260928 02:15:45.121353 25280 leader_election.cc:288] T 3c53e1d19c0e48b29c04acdd3fe8fd41 P 98febb23b62c463d9f1770c3c07481f3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ac4678909a0b419daf9990f634f7ff5e (127.6.235.65:34425), f5aa645bccb54161aa1c7645b66a917d (127.6.235.66:40613)
I20260928 02:15:45.127842 25256 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:15:45.121789 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.235.65:34425 (local address 127.6.235.67:44111)
0928 02:15:45.122079 (+ 290us) negotiation.cc:107] Waiting for socket to connect
0928 02:15:45.122102 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0928 02:15:45.122292 (+ 190us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:15:45.122700 (+ 408us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:15:45.122705 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:15:45.122735 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:15:45.122911 (+ 176us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:15:45.122919 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:15:45.123969 (+ 1050us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:15:45.123975 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:15:45.125587 (+ 1612us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:15:45.125603 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:15:45.126975 (+ 1372us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:15:45.126981 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:15:45.127172 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:15:45.127205 (+ 33us) client_negotiation.cc:769] Sending connection context
0928 02:15:45.127321 (+ 116us) client_negotiation.cc:240] Negotiation successful
0928 02:15:45.127498 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":187,"mutex_wait_us":86}
I20260928 02:15:45.127797 25266 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:15:45.121872 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.235.67:44111 (local address 127.6.235.65:34425)
0928 02:15:45.122061 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0928 02:15:45.122067 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 02:15:45.122203 (+ 136us) server_negotiation.cc:407] Connection header received
0928 02:15:45.122446 (+ 243us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:15:45.122450 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:15:45.122522 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:15:45.122623 (+ 101us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:15:45.123067 (+ 444us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:15:45.123812 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:15:45.125821 (+ 2009us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:15:45.126776 (+ 955us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:15:45.126984 (+ 208us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:15:45.127070 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0928 02:15:45.127352 (+ 282us) server_negotiation.cc:299] Negotiation successful
0928 02:15:45.127497 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260928 02:15:45.128249 25271 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:15:45.121897 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.235.67:55833 (local address 127.6.235.66:40613)
0928 02:15:45.122181 (+ 284us) server_negotiation.cc:206] Beginning negotiation
0928 02:15:45.122190 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 02:15:45.122864 (+ 674us) server_negotiation.cc:407] Connection header received
0928 02:15:45.123078 (+ 214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:15:45.123082 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:15:45.123153 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:15:45.123252 (+ 99us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:15:45.123783 (+ 531us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:15:45.124452 (+ 669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:15:45.126452 (+ 2000us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:15:45.127276 (+ 824us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:15:45.127466 (+ 190us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:15:45.127543 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0928 02:15:45.127840 (+ 297us) server_negotiation.cc:299] Negotiation successful
0928 02:15:45.127984 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260928 02:15:45.128382 25281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:15:45.121894 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.235.66:40613 (local address 127.6.235.67:55833)
0928 02:15:45.122734 (+ 840us) negotiation.cc:107] Waiting for socket to connect
0928 02:15:45.122748 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 02:15:45.122918 (+ 170us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:15:45.123264 (+ 346us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:15:45.123269 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:15:45.123292 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:15:45.123604 (+ 312us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:15:45.123618 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:15:45.124611 (+ 993us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:15:45.124616 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:15:45.126218 (+ 1602us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:15:45.126235 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:15:45.127465 (+ 1230us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:15:45.127475 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:15:45.127672 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:15:45.127695 (+ 23us) client_negotiation.cc:769] Sending connection context
0928 02:15:45.127800 (+ 105us) client_negotiation.cc:240] Negotiation successful
0928 02:15:45.127950 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":692,"mutex_wait_us":1,"thread_start_us":504,"threads_started":1}
I20260928 02:15:45.129421 25079 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c53e1d19c0e48b29c04acdd3fe8fd41" candidate_uuid: "98febb23b62c463d9f1770c3c07481f3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f5aa645bccb54161aa1c7645b66a917d" is_pre_election: true
I20260928 02:15:45.129400 24948 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c53e1d19c0e48b29c04acdd3fe8fd41" candidate_uuid: "98febb23b62c463d9f1770c3c07481f3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ac4678909a0b419daf9990f634f7ff5e" is_pre_election: true
I20260928 02:15:45.129959 24948 raft_consensus.cc:2454] T 3c53e1d19c0e48b29c04acdd3fe8fd41 P ac4678909a0b419daf9990f634f7ff5e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 98febb23b62c463d9f1770c3c07481f3 in term 0.
I20260928 02:15:45.129964 25079 raft_consensus.cc:2454] T 3c53e1d19c0e48b29c04acdd3fe8fd41 P f5aa645bccb54161aa1c7645b66a917d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 98febb23b62c463d9f1770c3c07481f3 in term 0.
I20260928 02:15:45.130667 25142 leader_election.cc:302] T 3c53e1d19c0e48b29c04acdd3fe8fd41 P 98febb23b62c463d9f1770c3c07481f3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 98febb23b62c463d9f1770c3c07481f3, f5aa645bccb54161aa1c7645b66a917d; no voters:
I20260928 02:15:45.131126 25280 raft_consensus.cc:2781] T 3c53e1d19c0e48b29c04acdd3fe8fd41 P 98febb23b62c463d9f1770c3c07481f3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 02:15:45.131295 25280 raft_consensus.cc:482] T 3c53e1d19c0e48b29c04acdd3fe8fd41 P 98febb23b62c463d9f1770c3c07481f3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 02:15:45.131410 25280 raft_consensus.cc:3037] T 3c53e1d19c0e48b29c04acdd3fe8fd41 P 98febb23b62c463d9f1770c3c07481f3 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:15:45.134222 25280 raft_consensus.cc:504] T 3c53e1d19c0e48b29c04acdd3fe8fd41 P 98febb23b62c463d9f1770c3c07481f3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac4678909a0b419daf9990f634f7ff5e" member_type: VOTER last_known_addr { host: "127.6.235.65" port: 34425 } } peers { permanent_uuid: "98febb23b62c463d9f1770c3c07481f3" member_type: VOTER last_known_addr { host: "127.6.235.67" port: 39559 } } peers { permanent_uuid: "f5aa645bccb54161aa1c7645b66a917d" member_type: VOTER last_known_addr { host: "127.6.235.66" port: 40613 } }
I20260928 02:15:45.134900 25280 leader_election.cc:288] T 3c53e1d19c0e48b29c04acdd3fe8fd41 P 98febb23b62c463d9f1770c3c07481f3 [CANDIDATE]: Term 1 election: Requested vote from peers ac4678909a0b419daf9990f634f7ff5e (127.6.235.65:34425), f5aa645bccb54161aa1c7645b66a917d (127.6.235.66:40613)
I20260928 02:15:45.135489 25079 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c53e1d19c0e48b29c04acdd3fe8fd41" candidate_uuid: "98febb23b62c463d9f1770c3c07481f3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f5aa645bccb54161aa1c7645b66a917d"
I20260928 02:15:45.135491 24948 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c53e1d19c0e48b29c04acdd3fe8fd41" candidate_uuid: "98febb23b62c463d9f1770c3c07481f3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ac4678909a0b419daf9990f634f7ff5e"
I20260928 02:15:45.135742 24948 raft_consensus.cc:3037] T 3c53e1d19c0e48b29c04acdd3fe8fd41 P ac4678909a0b419daf9990f634f7ff5e [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:15:45.135742 25079 raft_consensus.cc:3037] T 3c53e1d19c0e48b29c04acdd3fe8fd41 P f5aa645bccb54161aa1c7645b66a917d [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:15:45.139282 25079 raft_consensus.cc:2454] T 3c53e1d19c0e48b29c04acdd3fe8fd41 P f5aa645bccb54161aa1c7645b66a917d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 98febb23b62c463d9f1770c3c07481f3 in term 1.
I20260928 02:15:45.139279 24948 raft_consensus.cc:2454] T 3c53e1d19c0e48b29c04acdd3fe8fd41 P ac4678909a0b419daf9990f634f7ff5e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 98febb23b62c463d9f1770c3c07481f3 in term 1.
I20260928 02:15:45.139890 25145 leader_election.cc:302] T 3c53e1d19c0e48b29c04acdd3fe8fd41 P 98febb23b62c463d9f1770c3c07481f3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 98febb23b62c463d9f1770c3c07481f3, ac4678909a0b419daf9990f634f7ff5e, f5aa645bccb54161aa1c7645b66a917d; no voters:
I20260928 02:15:45.140453 25280 raft_consensus.cc:2781] T 3c53e1d19c0e48b29c04acdd3fe8fd41 P 98febb23b62c463d9f1770c3c07481f3 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:15:45.141249 25280 raft_consensus.cc:686] T 3c53e1d19c0e48b29c04acdd3fe8fd41 P 98febb23b62c463d9f1770c3c07481f3 [term 1 LEADER]: Becoming Leader. State: Replica: 98febb23b62c463d9f1770c3c07481f3, State: Running, Role: LEADER
I20260928 02:15:45.141803 25280 consensus_queue.cc:237] T 3c53e1d19c0e48b29c04acdd3fe8fd41 P 98febb23b62c463d9f1770c3c07481f3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac4678909a0b419daf9990f634f7ff5e" member_type: VOTER last_known_addr { host: "127.6.235.65" port: 34425 } } peers { permanent_uuid: "98febb23b62c463d9f1770c3c07481f3" member_type: VOTER last_known_addr { host: "127.6.235.67" port: 39559 } } peers { permanent_uuid: "f5aa645bccb54161aa1c7645b66a917d" member_type: VOTER last_known_addr { host: "127.6.235.66" port: 40613 } }
I20260928 02:15:45.147409 24806 catalog_manager.cc:5495] T 3c53e1d19c0e48b29c04acdd3fe8fd41 P 98febb23b62c463d9f1770c3c07481f3 reported cstate change: term changed from 0 to 1, leader changed from <none> to 98febb23b62c463d9f1770c3c07481f3 (127.6.235.67). New cstate: current_term: 1 leader_uuid: "98febb23b62c463d9f1770c3c07481f3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac4678909a0b419daf9990f634f7ff5e" member_type: VOTER last_known_addr { host: "127.6.235.65" port: 34425 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "98febb23b62c463d9f1770c3c07481f3" member_type: VOTER last_known_addr { host: "127.6.235.67" port: 39559 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f5aa645bccb54161aa1c7645b66a917d" member_type: VOTER last_known_addr { host: "127.6.235.66" port: 40613 } health_report { overall_health: UNKNOWN } } }
I20260928 02:15:45.170483 25269 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:15:45.164679 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49704 (local address 127.6.235.67:39559)
0928 02:15:45.164877 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0928 02:15:45.164884 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 02:15:45.165004 (+ 120us) server_negotiation.cc:407] Connection header received
0928 02:15:45.165298 (+ 294us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:15:45.165302 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:15:45.165380 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:15:45.165542 (+ 162us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 02:15:45.165956 (+ 414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:15:45.166696 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:15:45.168145 (+ 1449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:15:45.168592 (+ 447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:15:45.168736 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:15:45.169332 (+ 596us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 02:15:45.169872 (+ 540us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 02:15:45.170039 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0928 02:15:45.170122 (+ 83us) server_negotiation.cc:299] Negotiation successful
0928 02:15:45.170216 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260928 02:15:45.179764 25079 raft_consensus.cc:1261] T 3c53e1d19c0e48b29c04acdd3fe8fd41 P f5aa645bccb54161aa1c7645b66a917d [term 1 FOLLOWER]: Refusing update from remote peer 98febb23b62c463d9f1770c3c07481f3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 02:15:45.179764 24948 raft_consensus.cc:1261] T 3c53e1d19c0e48b29c04acdd3fe8fd41 P ac4678909a0b419daf9990f634f7ff5e [term 1 FOLLOWER]: Refusing update from remote peer 98febb23b62c463d9f1770c3c07481f3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 02:15:45.181180 25282 consensus_queue.cc:1035] T 3c53e1d19c0e48b29c04acdd3fe8fd41 P 98febb23b62c463d9f1770c3c07481f3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f5aa645bccb54161aa1c7645b66a917d" member_type: VOTER last_known_addr { host: "127.6.235.66" port: 40613 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 02:15:45.181648 25280 consensus_queue.cc:1035] T 3c53e1d19c0e48b29c04acdd3fe8fd41 P 98febb23b62c463d9f1770c3c07481f3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ac4678909a0b419daf9990f634f7ff5e" member_type: VOTER last_known_addr { host: "127.6.235.65" port: 34425 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 02:15:45.194906 25285 mvcc.cc:205] Tried to move back new op lower bound from 7334140908243439616 to 7334140908105981952. Current Snapshot: MvccSnapshot[applied={T|T < 7334140908243439616}]
I20260928 02:15:45.195724 25286 mvcc.cc:205] Tried to move back new op lower bound from 7334140908243439616 to 7334140908105981952. Current Snapshot: MvccSnapshot[applied={T|T < 7334140908243439616}]
I20260928 02:15:45.212086 24807 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:45516:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 02:15:45.212393 24807 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:45516:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 02:15:45.216398 24807 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 5cb923917cdd4dd8bd6ebdd8b7634d16: Sending DeleteTablet for 3 replicas of tablet 3c53e1d19c0e48b29c04acdd3fe8fd41
I20260928 02:15:45.217808 24928 tablet_service.cc:1514] Processing DeleteTablet for tablet 3c53e1d19c0e48b29c04acdd3fe8fd41 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:15:45 UTC) from {username='slave'} at 127.0.0.1:35510
I20260928 02:15:45.217938 25059 tablet_service.cc:1514] Processing DeleteTablet for tablet 3c53e1d19c0e48b29c04acdd3fe8fd41 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:15:45 UTC) from {username='slave'} at 127.0.0.1:34196
I20260928 02:15:45.217976 25189 tablet_service.cc:1514] Processing DeleteTablet for tablet 3c53e1d19c0e48b29c04acdd3fe8fd41 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:15:45 UTC) from {username='slave'} at 127.0.0.1:49688
I20260928 02:15:45.218909 25295 tablet_replica.cc:330] T 3c53e1d19c0e48b29c04acdd3fe8fd41 P ac4678909a0b419daf9990f634f7ff5e: stopping tablet replica
I20260928 02:15:45.219516 25295 raft_consensus.cc:2229] T 3c53e1d19c0e48b29c04acdd3fe8fd41 P ac4678909a0b419daf9990f634f7ff5e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 02:15:45.219769 7085 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskTlaUuc/build/asan/bin/kudu pid 24866
I20260928 02:15:45.220240 25295 raft_consensus.cc:2258] T 3c53e1d19c0e48b29c04acdd3fe8fd41 P ac4678909a0b419daf9990f634f7ff5e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 02:15:45.223107 25295 ts_tablet_manager.cc:1905] T 3c53e1d19c0e48b29c04acdd3fe8fd41 P ac4678909a0b419daf9990f634f7ff5e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 02:15:45.231688 25295 ts_tablet_manager.cc:1918] T 3c53e1d19c0e48b29c04acdd3fe8fd41 P ac4678909a0b419daf9990f634f7ff5e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 02:15:45.232280 25295 log.cc:1196] T 3c53e1d19c0e48b29c04acdd3fe8fd41 P ac4678909a0b419daf9990f634f7ff5e: Deleting WAL directory at /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790561432251112-7085-0/minicluster-data/ts-0/wal/wals/3c53e1d19c0e48b29c04acdd3fe8fd41
I20260928 02:15:45.233166 25295 ts_tablet_manager.cc:1939] T 3c53e1d19c0e48b29c04acdd3fe8fd41 P ac4678909a0b419daf9990f634f7ff5e: Deleting consensus metadata
I20260928 02:15:45.235854 24794 catalog_manager.cc:4841] TS ac4678909a0b419daf9990f634f7ff5e (127.6.235.65:34425): tablet 3c53e1d19c0e48b29c04acdd3fe8fd41 (table test-table [id=718138cca35742a29a2e1fcfa2824b7f]) successfully deleted
I20260928 02:15:45.245424 25297 tablet_replica.cc:330] T 3c53e1d19c0e48b29c04acdd3fe8fd41 P f5aa645bccb54161aa1c7645b66a917d: stopping tablet replica
I20260928 02:15:45.245723 25296 tablet_replica.cc:330] T 3c53e1d19c0e48b29c04acdd3fe8fd41 P 98febb23b62c463d9f1770c3c07481f3: stopping tablet replica
I20260928 02:15:45.245983 25297 raft_consensus.cc:2229] T 3c53e1d19c0e48b29c04acdd3fe8fd41 P f5aa645bccb54161aa1c7645b66a917d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 02:15:45.246435 25296 raft_consensus.cc:2229] T 3c53e1d19c0e48b29c04acdd3fe8fd41 P 98febb23b62c463d9f1770c3c07481f3 [term 1 LEADER]: Raft consensus shutting down.
I20260928 02:15:45.246609 25297 raft_consensus.cc:2258] T 3c53e1d19c0e48b29c04acdd3fe8fd41 P f5aa645bccb54161aa1c7645b66a917d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 02:15:45.247298 25296 raft_consensus.cc:2258] T 3c53e1d19c0e48b29c04acdd3fe8fd41 P 98febb23b62c463d9f1770c3c07481f3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 02:15:45.249413 25297 ts_tablet_manager.cc:1905] T 3c53e1d19c0e48b29c04acdd3fe8fd41 P f5aa645bccb54161aa1c7645b66a917d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 02:15:45.249444 25296 ts_tablet_manager.cc:1905] T 3c53e1d19c0e48b29c04acdd3fe8fd41 P 98febb23b62c463d9f1770c3c07481f3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 02:15:45.250080 25266 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:15:45.221281 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35518 (local address 127.6.235.65:34425)
0928 02:15:45.221503 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0928 02:15:45.221511 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 02:15:45.238315 (+ 16804us) server_negotiation.cc:407] Connection header received
0928 02:15:45.238749 (+ 434us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:15:45.238761 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:15:45.238941 (+ 180us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:15:45.239046 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:15:45.240014 (+ 968us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:15:45.241919 (+ 1905us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:15:45.245434 (+ 3515us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:15:45.246069 (+ 635us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:15:45.246203 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:15:45.248203 (+ 2000us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:15:45.248218 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:15:45.248235 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 02:15:45.249124 (+ 889us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:15:45.249189 (+ 65us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 02:15:45.249214 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:15:45.249337 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0928 02:15:45.249578 (+ 241us) server_negotiation.cc:299] Negotiation successful
0928 02:15:45.249726 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260928 02:15:45.251317 24988 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 02:15:45.258213 25297 ts_tablet_manager.cc:1918] T 3c53e1d19c0e48b29c04acdd3fe8fd41 P f5aa645bccb54161aa1c7645b66a917d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 02:15:45.258422 25297 log.cc:1196] T 3c53e1d19c0e48b29c04acdd3fe8fd41 P f5aa645bccb54161aa1c7645b66a917d: Deleting WAL directory at /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790561432251112-7085-0/minicluster-data/ts-1/wal/wals/3c53e1d19c0e48b29c04acdd3fe8fd41
I20260928 02:15:45.258847 25296 ts_tablet_manager.cc:1918] T 3c53e1d19c0e48b29c04acdd3fe8fd41 P 98febb23b62c463d9f1770c3c07481f3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 02:15:45.259057 25297 ts_tablet_manager.cc:1939] T 3c53e1d19c0e48b29c04acdd3fe8fd41 P f5aa645bccb54161aa1c7645b66a917d: Deleting consensus metadata
I20260928 02:15:45.259095 25296 log.cc:1196] T 3c53e1d19c0e48b29c04acdd3fe8fd41 P 98febb23b62c463d9f1770c3c07481f3: Deleting WAL directory at /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790561432251112-7085-0/minicluster-data/ts-2/wal/wals/3c53e1d19c0e48b29c04acdd3fe8fd41
I20260928 02:15:45.259704 25296 ts_tablet_manager.cc:1939] T 3c53e1d19c0e48b29c04acdd3fe8fd41 P 98febb23b62c463d9f1770c3c07481f3: Deleting consensus metadata
I20260928 02:15:45.261300 24791 catalog_manager.cc:4841] TS f5aa645bccb54161aa1c7645b66a917d (127.6.235.66:40613): tablet 3c53e1d19c0e48b29c04acdd3fe8fd41 (table test-table [id=718138cca35742a29a2e1fcfa2824b7f]) successfully deleted
I20260928 02:15:45.261687 24791 catalog_manager.cc:4841] TS 98febb23b62c463d9f1770c3c07481f3 (127.6.235.67:39559): tablet 3c53e1d19c0e48b29c04acdd3fe8fd41 (table test-table [id=718138cca35742a29a2e1fcfa2824b7f]) successfully deleted
I20260928 02:15:45.347110 7085 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTlaUuc/build/asan/bin/kudu with pid 24866
I20260928 02:15:45.366616 7085 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskTlaUuc/build/asan/bin/kudu pid 24997
I20260928 02:15:45.375032 25271 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:15:45.367553 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34212 (local address 127.6.235.66:40613)
0928 02:15:45.367768 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0928 02:15:45.367775 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 02:15:45.367876 (+ 101us) server_negotiation.cc:407] Connection header received
0928 02:15:45.368035 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:15:45.368041 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:15:45.368175 (+ 134us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:15:45.368299 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:15:45.368690 (+ 391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:15:45.369855 (+ 1165us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:15:45.371193 (+ 1338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:15:45.371877 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:15:45.372012 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:15:45.373297 (+ 1285us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:15:45.373310 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:15:45.373331 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 02:15:45.374196 (+ 865us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:15:45.374263 (+ 67us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 02:15:45.374295 (+ 32us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:15:45.374415 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0928 02:15:45.374531 (+ 116us) server_negotiation.cc:299] Negotiation successful
0928 02:15:45.374663 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260928 02:15:45.375908 25119 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 02:15:45.467904 7085 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTlaUuc/build/asan/bin/kudu with pid 24997
I20260928 02:15:45.488154 7085 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskTlaUuc/build/asan/bin/kudu pid 25127
I20260928 02:15:45.496805 25269 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:15:45.488998 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49708 (local address 127.6.235.67:39559)
0928 02:15:45.489195 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0928 02:15:45.489204 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 02:15:45.489323 (+ 119us) server_negotiation.cc:407] Connection header received
0928 02:15:45.489533 (+ 210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:15:45.489539 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:15:45.489636 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:15:45.489765 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:15:45.490181 (+ 416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:15:45.491059 (+ 878us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:15:45.492858 (+ 1799us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:15:45.493453 (+ 595us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:15:45.493629 (+ 176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:15:45.494959 (+ 1330us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:15:45.494970 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:15:45.494992 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 02:15:45.495926 (+ 934us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:15:45.495993 (+ 67us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 02:15:45.496027 (+ 34us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:15:45.496153 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0928 02:15:45.496307 (+ 154us) server_negotiation.cc:299] Negotiation successful
0928 02:15:45.496424 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260928 02:15:45.497584 25249 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 02:15:45.621490 7085 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTlaUuc/build/asan/bin/kudu with pid 25127
I20260928 02:15:45.641644 7085 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskTlaUuc/build/asan/bin/kudu pid 24776
I20260928 02:15:45.642908 24837 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 02:15:45.755636 7085 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTlaUuc/build/asan/bin/kudu with pid 24776
2026-09-28T02:15:45Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/1 (2890 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/2
Loading random data
Initializing database '/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790561432251112-7085-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 02:15:45 dist-test-slave-86gv krb5kdc[25305](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 02:15:45 dist-test-slave-86gv krb5kdc[25305](info): set up 2 sockets
Sep 28 02:15:45 dist-test-slave-86gv krb5kdc[25305](info): commencing operation
krb5kdc: starting...
W20260928 02:15:46.877555 7085 mini_kdc.cc:121] Time spent starting KDC: real 1.078s user 0.010s sys 0.021s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790561432251112-7085-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790561432251112-7085-0/krb5kdc/test-user.keytab.
Sep 28 02:15:47 dist-test-slave-86gv krb5kdc[25305](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561747, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T02:15:47Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T02:15:47Z Disabled control of system clock
WARNING: no policy specified for kudu/127.6.235.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.235.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.235.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790561432251112-7085-0/krb5kdc/kudu_127.6.235.126.keytab.
Entry for principal kudu/127.6.235.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790561432251112-7085-0/krb5kdc/kudu_127.6.235.126.keytab.
I20260928 02:15:47.160143 7085 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTlaUuc/build/asan/bin/kudu
/tmp/dist-test-taskTlaUuc/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790561432251112-7085-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790561432251112-7085-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790561432251112-7085-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790561432251112-7085-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.235.126:36837
--webserver_interface=127.6.235.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790561432251112-7085-0/krb5kdc/kudu_127.6.235.126.keytab
--principal=kudu/127.6.235.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.235.84:36953
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.235.126:36837
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790561432251112-7085-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790561432251112-7085-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790561432251112-7085-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:15:47.514485 25318 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:15:47.514871 25318 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:15:47.515033 25318 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:15:47.524204 25318 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 02:15:47.524377 25318 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:15:47.524482 25318 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:15:47.524591 25318 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 02:15:47.524677 25318 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 02:15:47.535189 25318 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.235.84:36953
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790561432251112-7085-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790561432251112-7085-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.235.126:36837
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790561432251112-7085-0/krb5kdc/kudu_127.6.235.126.keytab
--principal=kudu/127.6.235.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.235.126:36837
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790561432251112-7085-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.235.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790561432251112-7085-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 00:51:59 UTC on ccycloud.quasar-axgkwn.root.comops.site
build id 4676464
ASAN enabled
I20260928 02:15:47.536669 25318 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:15:47.538159 25318 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:15:47.550467 25326 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:15:47.550519 25324 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:15:47.550437 25323 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:15:47.551647 25318 server_base.cc:1034] running on GCE node
I20260928 02:15:47.552552 25318 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:15:47.554207 25318 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:15:47.555469 25318 hybrid_clock.cc:648] HybridClock initialized: now 1790561747555414 us; error 48 us; skew 500 ppm
I20260928 02:15:47.555857 25318 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 02:15:47 dist-test-slave-86gv krb5kdc[25305](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561747, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.235.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:15:47.559509 25318 init.cc:377] Logged in from keytab as kudu/127.6.235.126@KRBTEST.COM (short username kudu)
I20260928 02:15:47.561420 25318 webserver.cc:460] Webserver started at http://127.6.235.126:45617/ using document root <none> and password file <none>
I20260928 02:15:47.562162 25318 fs_manager.cc:359] Metadata directory not provided
I20260928 02:15:47.562311 25318 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:15:47.562631 25318 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:15:47.565924 25318 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790561432251112-7085-0/minicluster-data/master-0/data/instance:
uuid: "def059cc76c74dfa91d4d5bea5074934"
format_stamp: "Formatted at 2026-09-28 02:15:47 on dist-test-slave-86gv"
server_key: "8afb7c12d7bc99498e1d3f972d0143b4"
server_key_iv: "7beb53df772277f7a98b285fff9aab96"
server_key_version: "encryptionkey@0"
I20260928 02:15:47.566730 25318 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790561432251112-7085-0/minicluster-data/master-0/wal/instance:
uuid: "def059cc76c74dfa91d4d5bea5074934"
format_stamp: "Formatted at 2026-09-28 02:15:47 on dist-test-slave-86gv"
server_key: "8afb7c12d7bc99498e1d3f972d0143b4"
server_key_iv: "7beb53df772277f7a98b285fff9aab96"
server_key_version: "encryptionkey@0"
I20260928 02:15:47.572252 25318 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260928 02:15:47.576277 25333 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:15:47.577806 25318 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 02:15:47.578073 25318 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790561432251112-7085-0/minicluster-data/master-0/data,/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790561432251112-7085-0/minicluster-data/master-0/wal
uuid: "def059cc76c74dfa91d4d5bea5074934"
format_stamp: "Formatted at 2026-09-28 02:15:47 on dist-test-slave-86gv"
server_key: "8afb7c12d7bc99498e1d3f972d0143b4"
server_key_iv: "7beb53df772277f7a98b285fff9aab96"
server_key_version: "encryptionkey@0"
I20260928 02:15:47.578295 25318 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790561432251112-7085-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790561432251112-7085-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790561432251112-7085-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:15:47.604436 25318 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:15:47.609171 25318 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:15:47.609470 25318 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:15:47.629473 25318 rpc_server.cc:307] RPC server started. Bound to: 127.6.235.126:36837
I20260928 02:15:47.629520 25384 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.235.126:36837 every 8 connection(s)
I20260928 02:15:47.631218 25318 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790561432251112-7085-0/minicluster-data/master-0/data/info.pb
I20260928 02:15:47.635356 25385 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:15:47.639199 7085 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTlaUuc/build/asan/bin/kudu as pid 25318
I20260928 02:15:47.639421 7085 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790561432251112-7085-0/minicluster-data/master-0/wal/instance
I20260928 02:15:47.640023 7085 external_mini_cluster.cc:1428] Setting key a0d15638fd96b363a43715bd072b699e
I20260928 02:15:47.649623 25385 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P def059cc76c74dfa91d4d5bea5074934: Bootstrap starting.
I20260928 02:15:47.653872 25385 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P def059cc76c74dfa91d4d5bea5074934: Neither blocks nor log segments found. Creating new log.
I20260928 02:15:47.655361 25385 log.cc:824] T 00000000000000000000000000000000 P def059cc76c74dfa91d4d5bea5074934: Log is configured to *not* fsync() on all Append() calls
Sep 28 02:15:47 dist-test-slave-86gv krb5kdc[25305](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561747, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.235.126@KRBTEST.COM
I20260928 02:15:47.658914 25385 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P def059cc76c74dfa91d4d5bea5074934: No bootstrap required, opened a new log
I20260928 02:15:47.667271 25388 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:15:47.642173 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58078 (local address 127.6.235.126:36837)
0928 02:15:47.643085 (+ 912us) server_negotiation.cc:206] Beginning negotiation
0928 02:15:47.643099 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0928 02:15:47.643142 (+ 43us) server_negotiation.cc:407] Connection header received
0928 02:15:47.645073 (+ 1931us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:15:47.645103 (+ 30us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:15:47.645622 (+ 519us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:15:47.646011 (+ 389us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:15:47.647923 (+ 1912us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:15:47.649400 (+ 1477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:15:47.651560 (+ 2160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:15:47.652507 (+ 947us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:15:47.652670 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:15:47.658321 (+ 5651us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:15:47.658331 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:15:47.658348 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:15:47.658394 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:15:47.662330 (+ 3936us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:15:47.663273 (+ 943us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:15:47.663280 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:15:47.663288 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:15:47.663483 (+ 195us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:15:47.663948 (+ 465us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:15:47.663952 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:15:47.663954 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:15:47.664620 (+ 666us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:15:47.664860 (+ 240us) server_negotiation.cc:1035] Waiting for connection context
0928 02:15:47.665410 (+ 550us) server_negotiation.cc:299] Negotiation successful
0928 02:15:47.665902 (+ 492us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":690,"thread_start_us":457,"threads_started":1}
I20260928 02:15:47.668620 25385 raft_consensus.cc:348] T 00000000000000000000000000000000 P def059cc76c74dfa91d4d5bea5074934 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "def059cc76c74dfa91d4d5bea5074934" member_type: VOTER last_known_addr { host: "127.6.235.126" port: 36837 } }
I20260928 02:15:47.669171 25385 raft_consensus.cc:374] T 00000000000000000000000000000000 P def059cc76c74dfa91d4d5bea5074934 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:15:47.669299 25385 raft_consensus.cc:729] T 00000000000000000000000000000000 P def059cc76c74dfa91d4d5bea5074934 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: def059cc76c74dfa91d4d5bea5074934, State: Initialized, Role: FOLLOWER
I20260928 02:15:47.669900 25385 consensus_queue.cc:260] T 00000000000000000000000000000000 P def059cc76c74dfa91d4d5bea5074934 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "def059cc76c74dfa91d4d5bea5074934" member_type: VOTER last_known_addr { host: "127.6.235.126" port: 36837 } }
I20260928 02:15:47.670248 25385 raft_consensus.cc:388] T 00000000000000000000000000000000 P def059cc76c74dfa91d4d5bea5074934 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 02:15:47.670406 25385 raft_consensus.cc:482] T 00000000000000000000000000000000 P def059cc76c74dfa91d4d5bea5074934 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 02:15:47.670590 25385 raft_consensus.cc:3037] T 00000000000000000000000000000000 P def059cc76c74dfa91d4d5bea5074934 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:15:47.673038 25385 raft_consensus.cc:504] T 00000000000000000000000000000000 P def059cc76c74dfa91d4d5bea5074934 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "def059cc76c74dfa91d4d5bea5074934" member_type: VOTER last_known_addr { host: "127.6.235.126" port: 36837 } }
I20260928 02:15:47.673553 25385 leader_election.cc:302] T 00000000000000000000000000000000 P def059cc76c74dfa91d4d5bea5074934 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: def059cc76c74dfa91d4d5bea5074934; no voters:
I20260928 02:15:47.674249 25385 leader_election.cc:288] T 00000000000000000000000000000000 P def059cc76c74dfa91d4d5bea5074934 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 02:15:47.674402 25390 raft_consensus.cc:2781] T 00000000000000000000000000000000 P def059cc76c74dfa91d4d5bea5074934 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:15:47.675227 25390 raft_consensus.cc:686] T 00000000000000000000000000000000 P def059cc76c74dfa91d4d5bea5074934 [term 1 LEADER]: Becoming Leader. State: Replica: def059cc76c74dfa91d4d5bea5074934, State: Running, Role: LEADER
I20260928 02:15:47.675868 25390 consensus_queue.cc:237] T 00000000000000000000000000000000 P def059cc76c74dfa91d4d5bea5074934 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "def059cc76c74dfa91d4d5bea5074934" member_type: VOTER last_known_addr { host: "127.6.235.126" port: 36837 } }
I20260928 02:15:47.676512 25385 sys_catalog.cc:564] T 00000000000000000000000000000000 P def059cc76c74dfa91d4d5bea5074934 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 02:15:47.679450 25392 sys_catalog.cc:455] T 00000000000000000000000000000000 P def059cc76c74dfa91d4d5bea5074934 [sys.catalog]: SysCatalogTable state changed. Reason: New leader def059cc76c74dfa91d4d5bea5074934. Latest consensus state: current_term: 1 leader_uuid: "def059cc76c74dfa91d4d5bea5074934" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "def059cc76c74dfa91d4d5bea5074934" member_type: VOTER last_known_addr { host: "127.6.235.126" port: 36837 } } }
I20260928 02:15:47.679811 25392 sys_catalog.cc:458] T 00000000000000000000000000000000 P def059cc76c74dfa91d4d5bea5074934 [sys.catalog]: This master's current role is: LEADER
I20260928 02:15:47.680112 25391 sys_catalog.cc:455] T 00000000000000000000000000000000 P def059cc76c74dfa91d4d5bea5074934 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "def059cc76c74dfa91d4d5bea5074934" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "def059cc76c74dfa91d4d5bea5074934" member_type: VOTER last_known_addr { host: "127.6.235.126" port: 36837 } } }
I20260928 02:15:47.680511 25391 sys_catalog.cc:458] T 00000000000000000000000000000000 P def059cc76c74dfa91d4d5bea5074934 [sys.catalog]: This master's current role is: LEADER
I20260928 02:15:47.681396 25397 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 02:15:47.688227 25397 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 02:15:47.699795 25397 catalog_manager.cc:1347] Generated new cluster ID: 7b4a823e39de40ff979ea36d71b846d1
I20260928 02:15:47.699937 25397 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 02:15:47.713487 25397 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 02:15:47.715402 25397 catalog_manager.cc:1504] Loading token signing keys...
I20260928 02:15:47.722860 25397 catalog_manager.cc:5868] T 00000000000000000000000000000000 P def059cc76c74dfa91d4d5bea5074934: Generated new TSK 0
I20260928 02:15:47.723775 25397 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.6.235.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.235.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.235.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790561432251112-7085-0/krb5kdc/kudu_127.6.235.65.keytab.
Entry for principal kudu/127.6.235.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790561432251112-7085-0/krb5kdc/kudu_127.6.235.65.keytab.
I20260928 02:15:47.793227 7085 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTlaUuc/build/asan/bin/kudu
/tmp/dist-test-taskTlaUuc/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790561432251112-7085-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790561432251112-7085-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790561432251112-7085-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790561432251112-7085-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.235.65:0
--local_ip_for_outbound_sockets=127.6.235.65
--webserver_interface=127.6.235.65
--webserver_port=0
--tserver_master_addrs=127.6.235.126:36837
--keytab_file=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790561432251112-7085-0/krb5kdc/kudu_127.6.235.65.keytab
--principal=kudu/127.6.235.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.235.84:36953
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790561432251112-7085-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790561432251112-7085-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790561432251112-7085-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:15:48.153226 25411 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:15:48.153537 25411 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:15:48.153695 25411 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:15:48.161861 25411 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:15:48.161994 25411 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:15:48.162206 25411 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.235.65
I20260928 02:15:48.173949 25411 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.235.84:36953
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790561432251112-7085-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790561432251112-7085-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790561432251112-7085-0/krb5kdc/kudu_127.6.235.65.keytab
--principal=kudu/127.6.235.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.235.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790561432251112-7085-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.6.235.65
--webserver_port=0
--tserver_master_addrs=127.6.235.126:36837
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.235.65
--log_dir=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790561432251112-7085-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 00:51:59 UTC on ccycloud.quasar-axgkwn.root.comops.site
build id 4676464
ASAN enabled
I20260928 02:15:48.175505 25411 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:15:48.177093 25411 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:15:48.192075 25417 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:15:48.192167 25416 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:15:48.192150 25419 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:15:48.194090 25411 server_base.cc:1034] running on GCE node
I20260928 02:15:48.194854 25411 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:15:48.195976 25411 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:15:48.197232 25411 hybrid_clock.cc:648] HybridClock initialized: now 1790561748197177 us; error 52 us; skew 500 ppm
I20260928 02:15:48.197599 25411 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 02:15:48 dist-test-slave-86gv krb5kdc[25305](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561748, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.235.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:15:48.201372 25411 init.cc:377] Logged in from keytab as kudu/127.6.235.65@KRBTEST.COM (short username kudu)
I20260928 02:15:48.203311 25411 webserver.cc:460] Webserver started at http://127.6.235.65:41321/ using document root <none> and password file <none>
I20260928 02:15:48.204116 25411 fs_manager.cc:359] Metadata directory not provided
I20260928 02:15:48.204278 25411 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:15:48.204618 25411 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:15:48.208015 25411 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790561432251112-7085-0/minicluster-data/ts-0/data/instance:
uuid: "be6b5f3afc9548b199d7852d0c9fce3a"
format_stamp: "Formatted at 2026-09-28 02:15:48 on dist-test-slave-86gv"
server_key: "cb359926da186463af7009e8c2125fc9"
server_key_iv: "ec2c0a0ade921b2aac43e30fdaa2e28e"
server_key_version: "encryptionkey@0"
I20260928 02:15:48.208968 25411 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790561432251112-7085-0/minicluster-data/ts-0/wal/instance:
uuid: "be6b5f3afc9548b199d7852d0c9fce3a"
format_stamp: "Formatted at 2026-09-28 02:15:48 on dist-test-slave-86gv"
server_key: "cb359926da186463af7009e8c2125fc9"
server_key_iv: "ec2c0a0ade921b2aac43e30fdaa2e28e"
server_key_version: "encryptionkey@0"
I20260928 02:15:48.214838 25411 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.002s
I20260928 02:15:48.219295 25426 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:15:48.221002 25411 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 02:15:48.221227 25411 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790561432251112-7085-0/minicluster-data/ts-0/data,/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790561432251112-7085-0/minicluster-data/ts-0/wal
uuid: "be6b5f3afc9548b199d7852d0c9fce3a"
format_stamp: "Formatted at 2026-09-28 02:15:48 on dist-test-slave-86gv"
server_key: "cb359926da186463af7009e8c2125fc9"
server_key_iv: "ec2c0a0ade921b2aac43e30fdaa2e28e"
server_key_version: "encryptionkey@0"
I20260928 02:15:48.221478 25411 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790561432251112-7085-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790561432251112-7085-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790561432251112-7085-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:15:48.237483 25411 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:15:48.243221 25411 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:15:48.243566 25411 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:15:48.244711 25411 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:15:48.246312 25411 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:15:48.246433 25411 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:15:48.246596 25411 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:15:48.246677 25411 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:15:48.281596 25411 rpc_server.cc:307] RPC server started. Bound to: 127.6.235.65:42121
I20260928 02:15:48.281721 25538 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.235.65:42121 every 8 connection(s)
I20260928 02:15:48.283149 25411 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790561432251112-7085-0/minicluster-data/ts-0/data/info.pb
I20260928 02:15:48.285110 7085 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTlaUuc/build/asan/bin/kudu as pid 25411
I20260928 02:15:48.285329 7085 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790561432251112-7085-0/minicluster-data/ts-0/wal/instance
I20260928 02:15:48.285995 7085 external_mini_cluster.cc:1428] Setting key e11fb30cf0324e49855a23c2e83875e3
Sep 28 02:15:48 dist-test-slave-86gv krb5kdc[25305](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561748, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.235.65@KRBTEST.COM for kudu/127.6.235.126@KRBTEST.COM
I20260928 02:15:48.311933 25541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:15:48.286643 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.235.65:51165 (local address 127.6.235.126:36837)
0928 02:15:48.287245 (+ 602us) server_negotiation.cc:206] Beginning negotiation
0928 02:15:48.287295 (+ 50us) server_negotiation.cc:399] Waiting for connection header
0928 02:15:48.288534 (+ 1239us) server_negotiation.cc:407] Connection header received
0928 02:15:48.292027 (+ 3493us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:15:48.292032 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:15:48.292105 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:15:48.292207 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:15:48.294806 (+ 2599us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:15:48.295650 (+ 844us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:15:48.298126 (+ 2476us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:15:48.298583 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:15:48.298773 (+ 190us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:15:48.303551 (+ 4778us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:15:48.303562 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:15:48.303570 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:15:48.303637 (+ 67us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:15:48.307816 (+ 4179us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:15:48.308713 (+ 897us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:15:48.308723 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:15:48.308730 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:15:48.308935 (+ 205us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:15:48.309513 (+ 578us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:15:48.309522 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:15:48.309529 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:15:48.309960 (+ 431us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:15:48.310145 (+ 185us) server_negotiation.cc:1035] Waiting for connection context
0928 02:15:48.311044 (+ 899us) server_negotiation.cc:299] Negotiation successful
0928 02:15:48.311373 (+ 329us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":459,"thread_start_us":306,"threads_started":1}
I20260928 02:15:48.312625 25542 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:15:48.287363 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.235.126:36837 (local address 127.6.235.65:51165)
0928 02:15:48.288329 (+ 966us) negotiation.cc:107] Waiting for socket to connect
0928 02:15:48.288387 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0928 02:15:48.291600 (+ 3213us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:15:48.292559 (+ 959us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:15:48.292575 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:15:48.293562 (+ 987us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:15:48.294545 (+ 983us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:15:48.294580 (+ 35us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:15:48.295863 (+ 1283us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:15:48.295873 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:15:48.297932 (+ 2059us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:15:48.297946 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:15:48.298921 (+ 975us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:15:48.298932 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:15:48.299347 (+ 415us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:15:48.300215 (+ 868us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:15:48.300242 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:15:48.303228 (+ 2986us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:15:48.308032 (+ 4804us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:15:48.308041 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:15:48.308057 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:15:48.308540 (+ 483us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:15:48.309095 (+ 555us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:15:48.309100 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:15:48.309103 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:15:48.309352 (+ 249us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:15:48.310167 (+ 815us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:15:48.310174 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:15:48.310568 (+ 394us) client_negotiation.cc:769] Sending connection context
0928 02:15:48.310841 (+ 273us) client_negotiation.cc:240] Negotiation successful
0928 02:15:48.311363 (+ 522us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":596,"thread_start_us":328,"threads_started":1}
I20260928 02:15:48.314541 25539 heartbeater.cc:347] Connected to a master server at 127.6.235.126:36837
I20260928 02:15:48.314882 25539 heartbeater.cc:464] Registering TS with master...
I20260928 02:15:48.315677 25539 heartbeater.cc:511] Master 127.6.235.126:36837 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.6.235.66@KRBTEST.COM; defaulting to no policy
I20260928 02:15:48.318015 25350 ts_manager.cc:194] Registered new tserver with Master: be6b5f3afc9548b199d7852d0c9fce3a (127.6.235.65:42121)
I20260928 02:15:48.319638 25350 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.235.65@KRBTEST.COM'} at 127.6.235.65:51165
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.235.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.235.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790561432251112-7085-0/krb5kdc/kudu_127.6.235.66.keytab.
Entry for principal kudu/127.6.235.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790561432251112-7085-0/krb5kdc/kudu_127.6.235.66.keytab.
I20260928 02:15:48.357398 7085 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTlaUuc/build/asan/bin/kudu
/tmp/dist-test-taskTlaUuc/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790561432251112-7085-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790561432251112-7085-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790561432251112-7085-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790561432251112-7085-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.235.66:0
--local_ip_for_outbound_sockets=127.6.235.66
--webserver_interface=127.6.235.66
--webserver_port=0
--tserver_master_addrs=127.6.235.126:36837
--keytab_file=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790561432251112-7085-0/krb5kdc/kudu_127.6.235.66.keytab
--principal=kudu/127.6.235.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.235.84:36953
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790561432251112-7085-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790561432251112-7085-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790561432251112-7085-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:15:48.728329 25545 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:15:48.728631 25545 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:15:48.728794 25545 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:15:48.736338 25545 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:15:48.736469 25545 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:15:48.736666 25545 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.235.66
I20260928 02:15:48.747157 25545 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.235.84:36953
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790561432251112-7085-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790561432251112-7085-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790561432251112-7085-0/krb5kdc/kudu_127.6.235.66.keytab
--principal=kudu/127.6.235.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.235.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790561432251112-7085-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.6.235.66
--webserver_port=0
--tserver_master_addrs=127.6.235.126:36837
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.235.66
--log_dir=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790561432251112-7085-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 00:51:59 UTC on ccycloud.quasar-axgkwn.root.comops.site
build id 4676464
ASAN enabled
I20260928 02:15:48.748623 25545 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:15:48.750638 25545 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:15:48.765275 25550 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:15:48.765349 25553 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:15:48.765292 25551 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:15:48.766443 25545 server_base.cc:1034] running on GCE node
I20260928 02:15:48.767122 25545 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:15:48.768129 25545 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:15:48.769348 25545 hybrid_clock.cc:648] HybridClock initialized: now 1790561748769295 us; error 42 us; skew 500 ppm
I20260928 02:15:48.769690 25545 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 02:15:48 dist-test-slave-86gv krb5kdc[25305](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561748, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.235.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:15:48.780048 25545 init.cc:377] Logged in from keytab as kudu/127.6.235.66@KRBTEST.COM (short username kudu)
I20260928 02:15:48.782121 25545 webserver.cc:460] Webserver started at http://127.6.235.66:38637/ using document root <none> and password file <none>
I20260928 02:15:48.782934 25545 fs_manager.cc:359] Metadata directory not provided
I20260928 02:15:48.783093 25545 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:15:48.783384 25545 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:15:48.786492 25545 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790561432251112-7085-0/minicluster-data/ts-1/data/instance:
uuid: "4a9e49ebe1e14e89b6afc82a3f7a67dc"
format_stamp: "Formatted at 2026-09-28 02:15:48 on dist-test-slave-86gv"
server_key: "f0ed86f76ee9c32b781324a8d153e977"
server_key_iv: "6012783f8efcbbc88aed5d7fd50bd5db"
server_key_version: "encryptionkey@0"
I20260928 02:15:48.787266 25545 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790561432251112-7085-0/minicluster-data/ts-1/wal/instance:
uuid: "4a9e49ebe1e14e89b6afc82a3f7a67dc"
format_stamp: "Formatted at 2026-09-28 02:15:48 on dist-test-slave-86gv"
server_key: "f0ed86f76ee9c32b781324a8d153e977"
server_key_iv: "6012783f8efcbbc88aed5d7fd50bd5db"
server_key_version: "encryptionkey@0"
I20260928 02:15:48.792651 25545 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260928 02:15:48.796657 25560 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:15:48.798246 25545 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 02:15:48.798394 25545 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790561432251112-7085-0/minicluster-data/ts-1/data,/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790561432251112-7085-0/minicluster-data/ts-1/wal
uuid: "4a9e49ebe1e14e89b6afc82a3f7a67dc"
format_stamp: "Formatted at 2026-09-28 02:15:48 on dist-test-slave-86gv"
server_key: "f0ed86f76ee9c32b781324a8d153e977"
server_key_iv: "6012783f8efcbbc88aed5d7fd50bd5db"
server_key_version: "encryptionkey@0"
I20260928 02:15:48.798583 25545 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790561432251112-7085-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790561432251112-7085-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790561432251112-7085-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:15:48.811998 25545 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:15:48.816785 25545 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:15:48.817195 25545 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:15:48.818208 25545 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:15:48.819603 25545 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:15:48.819679 25545 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:15:48.819804 25545 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:15:48.819854 25545 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:15:48.853993 25545 rpc_server.cc:307] RPC server started. Bound to: 127.6.235.66:37759
I20260928 02:15:48.854097 25672 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.235.66:37759 every 8 connection(s)
I20260928 02:15:48.855463 25545 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790561432251112-7085-0/minicluster-data/ts-1/data/info.pb
I20260928 02:15:48.857214 7085 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTlaUuc/build/asan/bin/kudu as pid 25545
I20260928 02:15:48.857401 7085 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790561432251112-7085-0/minicluster-data/ts-1/wal/instance
I20260928 02:15:48.857920 7085 external_mini_cluster.cc:1428] Setting key dac7acdd44c3e90152390e82fb79c35d
Sep 28 02:15:48 dist-test-slave-86gv krb5kdc[25305](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561748, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.235.66@KRBTEST.COM for kudu/127.6.235.126@KRBTEST.COM
I20260928 02:15:48.881235 25675 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:15:48.858713 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.235.66:56707 (local address 127.6.235.126:36837)
0928 02:15:48.859426 (+ 713us) server_negotiation.cc:206] Beginning negotiation
0928 02:15:48.859432 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 02:15:48.860631 (+ 1199us) server_negotiation.cc:407] Connection header received
0928 02:15:48.862707 (+ 2076us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:15:48.862713 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:15:48.862812 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:15:48.862957 (+ 145us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:15:48.864882 (+ 1925us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:15:48.865679 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:15:48.867567 (+ 1888us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:15:48.868071 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:15:48.868286 (+ 215us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:15:48.872836 (+ 4550us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:15:48.872847 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:15:48.872856 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:15:48.872942 (+ 86us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:15:48.877271 (+ 4329us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:15:48.878233 (+ 962us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:15:48.878244 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:15:48.878250 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:15:48.878466 (+ 216us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:15:48.878955 (+ 489us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:15:48.878964 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:15:48.878971 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:15:48.879410 (+ 439us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:15:48.879581 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0928 02:15:48.880347 (+ 766us) server_negotiation.cc:299] Negotiation successful
0928 02:15:48.880635 (+ 288us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":589,"thread_start_us":467,"threads_started":1}
I20260928 02:15:48.881987 25676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:15:48.859465 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.235.126:36837 (local address 127.6.235.66:56707)
0928 02:15:48.860459 (+ 994us) negotiation.cc:107] Waiting for socket to connect
0928 02:15:48.860512 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0928 02:15:48.862357 (+ 1845us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:15:48.863253 (+ 896us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:15:48.863262 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:15:48.863868 (+ 606us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:15:48.864670 (+ 802us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:15:48.864693 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:15:48.865879 (+ 1186us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:15:48.865883 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:15:48.867363 (+ 1480us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:15:48.867379 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:15:48.868387 (+ 1008us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:15:48.868395 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:15:48.868680 (+ 285us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:15:48.869601 (+ 921us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:15:48.869625 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:15:48.872488 (+ 2863us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:15:48.877504 (+ 5016us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:15:48.877514 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:15:48.877530 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:15:48.878061 (+ 531us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:15:48.878614 (+ 553us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:15:48.878617 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:15:48.878620 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:15:48.878820 (+ 200us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:15:48.879550 (+ 730us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:15:48.879557 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:15:48.879936 (+ 379us) client_negotiation.cc:769] Sending connection context
0928 02:15:48.880189 (+ 253us) client_negotiation.cc:240] Negotiation successful
0928 02:15:48.880626 (+ 437us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":691,"thread_start_us":333,"threads_started":1}
I20260928 02:15:48.883739 25673 heartbeater.cc:347] Connected to a master server at 127.6.235.126:36837
I20260928 02:15:48.884081 25673 heartbeater.cc:464] Registering TS with master...
I20260928 02:15:48.884863 25673 heartbeater.cc:511] Master 127.6.235.126:36837 requested a full tablet report, sending...
I20260928 02:15:48.886615 25350 ts_manager.cc:194] Registered new tserver with Master: 4a9e49ebe1e14e89b6afc82a3f7a67dc (127.6.235.66:37759)
WARNING: no policy specified for kudu/127.6.235.67@KRBTEST.COM; defaulting to no policy
I20260928 02:15:48.887840 25350 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.235.66@KRBTEST.COM'} at 127.6.235.66:56707
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.235.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.235.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790561432251112-7085-0/krb5kdc/kudu_127.6.235.67.keytab.
Entry for principal kudu/127.6.235.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790561432251112-7085-0/krb5kdc/kudu_127.6.235.67.keytab.
I20260928 02:15:48.927349 7085 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTlaUuc/build/asan/bin/kudu
/tmp/dist-test-taskTlaUuc/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790561432251112-7085-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790561432251112-7085-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790561432251112-7085-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790561432251112-7085-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.235.67:0
--local_ip_for_outbound_sockets=127.6.235.67
--webserver_interface=127.6.235.67
--webserver_port=0
--tserver_master_addrs=127.6.235.126:36837
--keytab_file=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790561432251112-7085-0/krb5kdc/kudu_127.6.235.67.keytab
--principal=kudu/127.6.235.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.235.84:36953
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790561432251112-7085-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790561432251112-7085-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790561432251112-7085-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:15:49.292830 25679 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:15:49.293286 25679 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:15:49.293467 25679 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:15:49.301265 25679 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:15:49.301445 25679 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:15:49.301638 25679 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.235.67
I20260928 02:15:49.312098 25679 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.235.84:36953
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790561432251112-7085-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790561432251112-7085-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790561432251112-7085-0/krb5kdc/kudu_127.6.235.67.keytab
--principal=kudu/127.6.235.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.235.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790561432251112-7085-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.6.235.67
--webserver_port=0
--tserver_master_addrs=127.6.235.126:36837
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.235.67
--log_dir=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790561432251112-7085-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 00:51:59 UTC on ccycloud.quasar-axgkwn.root.comops.site
build id 4676464
ASAN enabled
I20260928 02:15:49.313594 25679 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:15:49.315150 25679 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260928 02:15:49.323073 25539 heartbeater.cc:503] Master 127.6.235.126:36837 was elected leader, sending a full tablet report...
W20260928 02:15:49.332267 25687 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:15:49.332352 25684 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:15:49.332383 25685 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:15:49.333236 25679 server_base.cc:1034] running on GCE node
I20260928 02:15:49.333935 25679 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:15:49.335084 25679 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:15:49.336329 25679 hybrid_clock.cc:648] HybridClock initialized: now 1790561749336262 us; error 74 us; skew 500 ppm
I20260928 02:15:49.336692 25679 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 02:15:49 dist-test-slave-86gv krb5kdc[25305](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561749, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.235.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:15:49.340116 25679 init.cc:377] Logged in from keytab as kudu/127.6.235.67@KRBTEST.COM (short username kudu)
I20260928 02:15:49.342067 25679 webserver.cc:460] Webserver started at http://127.6.235.67:33219/ using document root <none> and password file <none>
I20260928 02:15:49.342769 25679 fs_manager.cc:359] Metadata directory not provided
I20260928 02:15:49.342871 25679 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:15:49.343164 25679 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:15:49.346341 25679 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790561432251112-7085-0/minicluster-data/ts-2/data/instance:
uuid: "4c8ae0528987411988e503b342f1be5d"
format_stamp: "Formatted at 2026-09-28 02:15:49 on dist-test-slave-86gv"
server_key: "9cccdce1681c53661ae37e60f9327f5f"
server_key_iv: "ddbf27e2d7515d09d206941bcebc0bfa"
server_key_version: "encryptionkey@0"
I20260928 02:15:49.347154 25679 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790561432251112-7085-0/minicluster-data/ts-2/wal/instance:
uuid: "4c8ae0528987411988e503b342f1be5d"
format_stamp: "Formatted at 2026-09-28 02:15:49 on dist-test-slave-86gv"
server_key: "9cccdce1681c53661ae37e60f9327f5f"
server_key_iv: "ddbf27e2d7515d09d206941bcebc0bfa"
server_key_version: "encryptionkey@0"
I20260928 02:15:49.353107 25679 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.000s
I20260928 02:15:49.357100 25694 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:15:49.358814 25679 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260928 02:15:49.358984 25679 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790561432251112-7085-0/minicluster-data/ts-2/data,/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790561432251112-7085-0/minicluster-data/ts-2/wal
uuid: "4c8ae0528987411988e503b342f1be5d"
format_stamp: "Formatted at 2026-09-28 02:15:49 on dist-test-slave-86gv"
server_key: "9cccdce1681c53661ae37e60f9327f5f"
server_key_iv: "ddbf27e2d7515d09d206941bcebc0bfa"
server_key_version: "encryptionkey@0"
I20260928 02:15:49.359189 25679 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790561432251112-7085-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790561432251112-7085-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790561432251112-7085-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:15:49.383412 25679 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:15:49.388295 25679 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:15:49.388568 25679 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:15:49.389595 25679 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:15:49.391145 25679 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:15:49.391222 25679 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:15:49.391311 25679 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:15:49.391355 25679 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:15:49.428037 25679 rpc_server.cc:307] RPC server started. Bound to: 127.6.235.67:46293
I20260928 02:15:49.428229 25806 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.235.67:46293 every 8 connection(s)
I20260928 02:15:49.429560 25679 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790561432251112-7085-0/minicluster-data/ts-2/data/info.pb
I20260928 02:15:49.436760 7085 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTlaUuc/build/asan/bin/kudu as pid 25679
I20260928 02:15:49.436985 7085 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790561432251112-7085-0/minicluster-data/ts-2/wal/instance
I20260928 02:15:49.437474 7085 external_mini_cluster.cc:1428] Setting key b6e6f6cb4236794c30c9544ad3185575
Sep 28 02:15:49 dist-test-slave-86gv krb5kdc[25305](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561749, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.235.67@KRBTEST.COM for kudu/127.6.235.126@KRBTEST.COM
I20260928 02:15:49.453598 25809 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:15:49.432449 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.235.67:43819 (local address 127.6.235.126:36837)
0928 02:15:49.433037 (+ 588us) server_negotiation.cc:206] Beginning negotiation
0928 02:15:49.433044 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 02:15:49.434136 (+ 1092us) server_negotiation.cc:407] Connection header received
0928 02:15:49.436163 (+ 2027us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:15:49.436168 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:15:49.436292 (+ 124us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:15:49.436440 (+ 148us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:15:49.438557 (+ 2117us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:15:49.439861 (+ 1304us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:15:49.441944 (+ 2083us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:15:49.442453 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:15:49.442570 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:15:49.446563 (+ 3993us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:15:49.446570 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:15:49.446573 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:15:49.446611 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:15:49.449598 (+ 2987us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:15:49.450405 (+ 807us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:15:49.450410 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:15:49.450412 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:15:49.450572 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:15:49.451207 (+ 635us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:15:49.451216 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:15:49.451222 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:15:49.451635 (+ 413us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:15:49.451805 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0928 02:15:49.452842 (+ 1037us) server_negotiation.cc:299] Negotiation successful
0928 02:15:49.453228 (+ 386us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":409,"thread_start_us":248,"threads_started":1}
I20260928 02:15:49.454653 25810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:15:49.433160 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.235.126:36837 (local address 127.6.235.67:43819)
0928 02:15:49.433952 (+ 792us) negotiation.cc:107] Waiting for socket to connect
0928 02:15:49.433994 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0928 02:15:49.435741 (+ 1747us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:15:49.436732 (+ 991us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:15:49.436742 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:15:49.437499 (+ 757us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:15:49.438343 (+ 844us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:15:49.438367 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:15:49.440045 (+ 1678us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:15:49.440051 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:15:49.441774 (+ 1723us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:15:49.441790 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:15:49.442749 (+ 959us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:15:49.442756 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:15:49.442971 (+ 215us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:15:49.443745 (+ 774us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:15:49.443769 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:15:49.446294 (+ 2525us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:15:49.449748 (+ 3454us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:15:49.449755 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:15:49.449771 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:15:49.450230 (+ 459us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:15:49.450737 (+ 507us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:15:49.450742 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:15:49.450744 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:15:49.450948 (+ 204us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:15:49.451827 (+ 879us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:15:49.451836 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:15:49.452293 (+ 457us) client_negotiation.cc:769] Sending connection context
0928 02:15:49.452716 (+ 423us) client_negotiation.cc:240] Negotiation successful
0928 02:15:49.453324 (+ 608us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":532,"thread_start_us":291,"threads_started":1}
I20260928 02:15:49.456342 25807 heartbeater.cc:347] Connected to a master server at 127.6.235.126:36837
I20260928 02:15:49.456697 25807 heartbeater.cc:464] Registering TS with master...
I20260928 02:15:49.457535 25807 heartbeater.cc:511] Master 127.6.235.126:36837 requested a full tablet report, sending...
I20260928 02:15:49.459228 25350 ts_manager.cc:194] Registered new tserver with Master: 4c8ae0528987411988e503b342f1be5d (127.6.235.67:46293)
I20260928 02:15:49.460080 25350 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.235.67@KRBTEST.COM'} at 127.6.235.67:43819
I20260928 02:15:49.466658 7085 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 02:15:49.486397 25809 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:15:49.471665 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58094 (local address 127.6.235.126:36837)
0928 02:15:49.471924 (+ 259us) server_negotiation.cc:206] Beginning negotiation
0928 02:15:49.471930 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 02:15:49.472234 (+ 304us) server_negotiation.cc:407] Connection header received
0928 02:15:49.472460 (+ 226us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:15:49.472465 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:15:49.472536 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:15:49.472656 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:15:49.474251 (+ 1595us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:15:49.474973 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:15:49.476352 (+ 1379us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:15:49.476841 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:15:49.477016 (+ 175us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:15:49.480157 (+ 3141us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:15:49.480168 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:15:49.480175 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:15:49.480224 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:15:49.483319 (+ 3095us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:15:49.484296 (+ 977us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:15:49.484300 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:15:49.484303 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:15:49.484424 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:15:49.484976 (+ 552us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:15:49.484980 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:15:49.484984 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:15:49.485337 (+ 353us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:15:49.485549 (+ 212us) server_negotiation.cc:1035] Waiting for connection context
0928 02:15:49.485970 (+ 421us) server_negotiation.cc:299] Negotiation successful
0928 02:15:49.486162 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260928 02:15:49.491802 25350 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58094:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 02:15:49.499389 25350 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 02:15:49.525519 25821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:15:49.513936 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.235.66:37759 (local address 127.0.0.1:52714)
0928 02:15:49.516281 (+ 2345us) negotiation.cc:107] Waiting for socket to connect
0928 02:15:49.516302 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0928 02:15:49.516417 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:15:49.517520 (+ 1103us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:15:49.517528 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:15:49.517560 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:15:49.517958 (+ 398us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:15:49.517971 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:15:49.520022 (+ 2051us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:15:49.520028 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:15:49.522244 (+ 2216us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:15:49.522257 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:15:49.524375 (+ 2118us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:15:49.524381 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:15:49.524608 (+ 227us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:15:49.524678 (+ 70us) client_negotiation.cc:769] Sending connection context
0928 02:15:49.524930 (+ 252us) client_negotiation.cc:240] Negotiation successful
0928 02:15:49.525177 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2225,"mutex_wait_us":30,"thread_start_us":2082,"threads_started":1}
I20260928 02:15:49.526494 25820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:15:49.513773 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.235.65:42121 (local address 127.0.0.1:58100)
0928 02:15:49.514959 (+ 1186us) negotiation.cc:107] Waiting for socket to connect
0928 02:15:49.515023 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0928 02:15:49.515243 (+ 220us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:15:49.518960 (+ 3717us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:15:49.518965 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:15:49.518990 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:15:49.519389 (+ 399us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:15:49.519400 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:15:49.521544 (+ 2144us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:15:49.521550 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:15:49.523709 (+ 2159us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:15:49.523723 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:15:49.525697 (+ 1974us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:15:49.525705 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:15:49.525907 (+ 202us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:15:49.525940 (+ 33us) client_negotiation.cc:769] Sending connection context
0928 02:15:49.526034 (+ 94us) client_negotiation.cc:240] Negotiation successful
0928 02:15:49.526197 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":879,"mutex_wait_us":20,"thread_start_us":715,"threads_started":1}
I20260928 02:15:49.526633 25822 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:15:49.513932 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58100 (local address 127.6.235.65:42121)
0928 02:15:49.518143 (+ 4211us) server_negotiation.cc:206] Beginning negotiation
0928 02:15:49.518151 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 02:15:49.518179 (+ 28us) server_negotiation.cc:407] Connection header received
0928 02:15:49.518276 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:15:49.518285 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:15:49.518730 (+ 445us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:15:49.519595 (+ 865us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:15:49.519839 (+ 244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:15:49.521392 (+ 1553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:15:49.523907 (+ 2515us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:15:49.525468 (+ 1561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:15:49.525610 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:15:49.525846 (+ 236us) server_negotiation.cc:1035] Waiting for connection context
0928 02:15:49.526109 (+ 263us) server_negotiation.cc:299] Negotiation successful
0928 02:15:49.526294 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4029,"thread_start_us":3869,"threads_started":1}
I20260928 02:15:49.526218 25824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:15:49.515569 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.235.67:46293 (local address 127.0.0.1:53074)
0928 02:15:49.516963 (+ 1394us) negotiation.cc:107] Waiting for socket to connect
0928 02:15:49.516985 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0928 02:15:49.517102 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:15:49.518329 (+ 1227us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:15:49.518333 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:15:49.518357 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:15:49.518812 (+ 455us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:15:49.518824 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:15:49.520635 (+ 1811us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:15:49.520642 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:15:49.522805 (+ 2163us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:15:49.522820 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:15:49.524956 (+ 2136us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:15:49.524963 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:15:49.525138 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:15:49.525178 (+ 40us) client_negotiation.cc:769] Sending connection context
0928 02:15:49.525265 (+ 87us) client_negotiation.cc:240] Negotiation successful
0928 02:15:49.525407 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1265,"thread_start_us":1137,"threads_started":1}
I20260928 02:15:49.527161 25825 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:15:49.516232 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53074 (local address 127.6.235.67:46293)
0928 02:15:49.517734 (+ 1502us) server_negotiation.cc:206] Beginning negotiation
0928 02:15:49.517741 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 02:15:49.517774 (+ 33us) server_negotiation.cc:407] Connection header received
0928 02:15:49.517874 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:15:49.517882 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:15:49.518139 (+ 257us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:15:49.518343 (+ 204us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:15:49.519044 (+ 701us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:15:49.520486 (+ 1442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:15:49.523006 (+ 2520us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:15:49.524538 (+ 1532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:15:49.524684 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:15:49.526235 (+ 1551us) server_negotiation.cc:1035] Waiting for connection context
0928 02:15:49.526304 (+ 69us) server_negotiation.cc:299] Negotiation successful
0928 02:15:49.526474 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1289,"thread_start_us":1088,"threads_started":1}
I20260928 02:15:49.527742 25823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:15:49.514511 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52714 (local address 127.6.235.66:37759)
0928 02:15:49.515849 (+ 1338us) server_negotiation.cc:206] Beginning negotiation
0928 02:15:49.515863 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0928 02:15:49.516540 (+ 677us) server_negotiation.cc:407] Connection header received
0928 02:15:49.516657 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:15:49.516668 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:15:49.516983 (+ 315us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:15:49.517195 (+ 212us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:15:49.518138 (+ 943us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:15:49.519842 (+ 1704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:15:49.522413 (+ 2571us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:15:49.523879 (+ 1466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:15:49.524019 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:15:49.524232 (+ 213us) server_negotiation.cc:1035] Waiting for connection context
0928 02:15:49.527131 (+ 2899us) server_negotiation.cc:299] Negotiation successful
0928 02:15:49.527367 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1159,"thread_start_us":1025,"threads_started":1}
I20260928 02:15:49.531513 25742 tablet_service.cc:1467] Processing CreateTablet for tablet 30e8ef1cc9354ebcb66b7dee9fbf91f2 (DEFAULT_TABLE table=test-table [id=5e3593e2b2a14c9e88f56481f4d8fb84]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:15:49.531565 25608 tablet_service.cc:1467] Processing CreateTablet for tablet 30e8ef1cc9354ebcb66b7dee9fbf91f2 (DEFAULT_TABLE table=test-table [id=5e3593e2b2a14c9e88f56481f4d8fb84]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:15:49.532879 25742 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 30e8ef1cc9354ebcb66b7dee9fbf91f2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:15:49.532999 25608 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 30e8ef1cc9354ebcb66b7dee9fbf91f2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:15:49.533849 25474 tablet_service.cc:1467] Processing CreateTablet for tablet 30e8ef1cc9354ebcb66b7dee9fbf91f2 (DEFAULT_TABLE table=test-table [id=5e3593e2b2a14c9e88f56481f4d8fb84]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:15:49.535446 25474 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 30e8ef1cc9354ebcb66b7dee9fbf91f2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:15:49.545449 25826 tablet_bootstrap.cc:492] T 30e8ef1cc9354ebcb66b7dee9fbf91f2 P 4c8ae0528987411988e503b342f1be5d: Bootstrap starting.
I20260928 02:15:49.548270 25827 tablet_bootstrap.cc:492] T 30e8ef1cc9354ebcb66b7dee9fbf91f2 P 4a9e49ebe1e14e89b6afc82a3f7a67dc: Bootstrap starting.
I20260928 02:15:49.550689 25828 tablet_bootstrap.cc:492] T 30e8ef1cc9354ebcb66b7dee9fbf91f2 P be6b5f3afc9548b199d7852d0c9fce3a: Bootstrap starting.
I20260928 02:15:49.550768 25826 tablet_bootstrap.cc:654] T 30e8ef1cc9354ebcb66b7dee9fbf91f2 P 4c8ae0528987411988e503b342f1be5d: Neither blocks nor log segments found. Creating new log.
I20260928 02:15:49.552323 25827 tablet_bootstrap.cc:654] T 30e8ef1cc9354ebcb66b7dee9fbf91f2 P 4a9e49ebe1e14e89b6afc82a3f7a67dc: Neither blocks nor log segments found. Creating new log.
I20260928 02:15:49.552623 25826 log.cc:824] T 30e8ef1cc9354ebcb66b7dee9fbf91f2 P 4c8ae0528987411988e503b342f1be5d: Log is configured to *not* fsync() on all Append() calls
I20260928 02:15:49.553596 25827 log.cc:824] T 30e8ef1cc9354ebcb66b7dee9fbf91f2 P 4a9e49ebe1e14e89b6afc82a3f7a67dc: Log is configured to *not* fsync() on all Append() calls
I20260928 02:15:49.553836 25828 tablet_bootstrap.cc:654] T 30e8ef1cc9354ebcb66b7dee9fbf91f2 P be6b5f3afc9548b199d7852d0c9fce3a: Neither blocks nor log segments found. Creating new log.
I20260928 02:15:49.555051 25828 log.cc:824] T 30e8ef1cc9354ebcb66b7dee9fbf91f2 P be6b5f3afc9548b199d7852d0c9fce3a: Log is configured to *not* fsync() on all Append() calls
I20260928 02:15:49.555927 25826 tablet_bootstrap.cc:492] T 30e8ef1cc9354ebcb66b7dee9fbf91f2 P 4c8ae0528987411988e503b342f1be5d: No bootstrap required, opened a new log
I20260928 02:15:49.556239 25826 ts_tablet_manager.cc:1397] T 30e8ef1cc9354ebcb66b7dee9fbf91f2 P 4c8ae0528987411988e503b342f1be5d: Time spent bootstrapping tablet: real 0.011s user 0.006s sys 0.000s
I20260928 02:15:49.556715 25827 tablet_bootstrap.cc:492] T 30e8ef1cc9354ebcb66b7dee9fbf91f2 P 4a9e49ebe1e14e89b6afc82a3f7a67dc: No bootstrap required, opened a new log
I20260928 02:15:49.557021 25827 ts_tablet_manager.cc:1397] T 30e8ef1cc9354ebcb66b7dee9fbf91f2 P 4a9e49ebe1e14e89b6afc82a3f7a67dc: Time spent bootstrapping tablet: real 0.009s user 0.008s sys 0.000s
I20260928 02:15:49.557869 25828 tablet_bootstrap.cc:492] T 30e8ef1cc9354ebcb66b7dee9fbf91f2 P be6b5f3afc9548b199d7852d0c9fce3a: No bootstrap required, opened a new log
I20260928 02:15:49.558120 25828 ts_tablet_manager.cc:1397] T 30e8ef1cc9354ebcb66b7dee9fbf91f2 P be6b5f3afc9548b199d7852d0c9fce3a: Time spent bootstrapping tablet: real 0.008s user 0.005s sys 0.001s
I20260928 02:15:49.565960 25828 raft_consensus.cc:348] T 30e8ef1cc9354ebcb66b7dee9fbf91f2 P be6b5f3afc9548b199d7852d0c9fce3a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be6b5f3afc9548b199d7852d0c9fce3a" member_type: VOTER last_known_addr { host: "127.6.235.65" port: 42121 } } peers { permanent_uuid: "4a9e49ebe1e14e89b6afc82a3f7a67dc" member_type: VOTER last_known_addr { host: "127.6.235.66" port: 37759 } } peers { permanent_uuid: "4c8ae0528987411988e503b342f1be5d" member_type: VOTER last_known_addr { host: "127.6.235.67" port: 46293 } }
I20260928 02:15:49.566383 25828 raft_consensus.cc:374] T 30e8ef1cc9354ebcb66b7dee9fbf91f2 P be6b5f3afc9548b199d7852d0c9fce3a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:15:49.566478 25828 raft_consensus.cc:729] T 30e8ef1cc9354ebcb66b7dee9fbf91f2 P be6b5f3afc9548b199d7852d0c9fce3a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: be6b5f3afc9548b199d7852d0c9fce3a, State: Initialized, Role: FOLLOWER
I20260928 02:15:49.567152 25828 consensus_queue.cc:260] T 30e8ef1cc9354ebcb66b7dee9fbf91f2 P be6b5f3afc9548b199d7852d0c9fce3a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be6b5f3afc9548b199d7852d0c9fce3a" member_type: VOTER last_known_addr { host: "127.6.235.65" port: 42121 } } peers { permanent_uuid: "4a9e49ebe1e14e89b6afc82a3f7a67dc" member_type: VOTER last_known_addr { host: "127.6.235.66" port: 37759 } } peers { permanent_uuid: "4c8ae0528987411988e503b342f1be5d" member_type: VOTER last_known_addr { host: "127.6.235.67" port: 46293 } }
I20260928 02:15:49.567545 25826 raft_consensus.cc:348] T 30e8ef1cc9354ebcb66b7dee9fbf91f2 P 4c8ae0528987411988e503b342f1be5d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be6b5f3afc9548b199d7852d0c9fce3a" member_type: VOTER last_known_addr { host: "127.6.235.65" port: 42121 } } peers { permanent_uuid: "4a9e49ebe1e14e89b6afc82a3f7a67dc" member_type: VOTER last_known_addr { host: "127.6.235.66" port: 37759 } } peers { permanent_uuid: "4c8ae0528987411988e503b342f1be5d" member_type: VOTER last_known_addr { host: "127.6.235.67" port: 46293 } }
I20260928 02:15:49.567624 25827 raft_consensus.cc:348] T 30e8ef1cc9354ebcb66b7dee9fbf91f2 P 4a9e49ebe1e14e89b6afc82a3f7a67dc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be6b5f3afc9548b199d7852d0c9fce3a" member_type: VOTER last_known_addr { host: "127.6.235.65" port: 42121 } } peers { permanent_uuid: "4a9e49ebe1e14e89b6afc82a3f7a67dc" member_type: VOTER last_known_addr { host: "127.6.235.66" port: 37759 } } peers { permanent_uuid: "4c8ae0528987411988e503b342f1be5d" member_type: VOTER last_known_addr { host: "127.6.235.67" port: 46293 } }
I20260928 02:15:49.568037 25827 raft_consensus.cc:374] T 30e8ef1cc9354ebcb66b7dee9fbf91f2 P 4a9e49ebe1e14e89b6afc82a3f7a67dc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:15:49.568024 25826 raft_consensus.cc:374] T 30e8ef1cc9354ebcb66b7dee9fbf91f2 P 4c8ae0528987411988e503b342f1be5d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:15:49.568138 25826 raft_consensus.cc:729] T 30e8ef1cc9354ebcb66b7dee9fbf91f2 P 4c8ae0528987411988e503b342f1be5d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4c8ae0528987411988e503b342f1be5d, State: Initialized, Role: FOLLOWER
I20260928 02:15:49.568138 25827 raft_consensus.cc:729] T 30e8ef1cc9354ebcb66b7dee9fbf91f2 P 4a9e49ebe1e14e89b6afc82a3f7a67dc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4a9e49ebe1e14e89b6afc82a3f7a67dc, State: Initialized, Role: FOLLOWER
I20260928 02:15:49.568929 25826 consensus_queue.cc:260] T 30e8ef1cc9354ebcb66b7dee9fbf91f2 P 4c8ae0528987411988e503b342f1be5d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be6b5f3afc9548b199d7852d0c9fce3a" member_type: VOTER last_known_addr { host: "127.6.235.65" port: 42121 } } peers { permanent_uuid: "4a9e49ebe1e14e89b6afc82a3f7a67dc" member_type: VOTER last_known_addr { host: "127.6.235.66" port: 37759 } } peers { permanent_uuid: "4c8ae0528987411988e503b342f1be5d" member_type: VOTER last_known_addr { host: "127.6.235.67" port: 46293 } }
I20260928 02:15:49.568985 25827 consensus_queue.cc:260] T 30e8ef1cc9354ebcb66b7dee9fbf91f2 P 4a9e49ebe1e14e89b6afc82a3f7a67dc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be6b5f3afc9548b199d7852d0c9fce3a" member_type: VOTER last_known_addr { host: "127.6.235.65" port: 42121 } } peers { permanent_uuid: "4a9e49ebe1e14e89b6afc82a3f7a67dc" member_type: VOTER last_known_addr { host: "127.6.235.66" port: 37759 } } peers { permanent_uuid: "4c8ae0528987411988e503b342f1be5d" member_type: VOTER last_known_addr { host: "127.6.235.67" port: 46293 } }
I20260928 02:15:49.573369 25673 heartbeater.cc:503] Master 127.6.235.126:36837 was elected leader, sending a full tablet report...
I20260928 02:15:49.576969 25827 ts_tablet_manager.cc:1428] T 30e8ef1cc9354ebcb66b7dee9fbf91f2 P 4a9e49ebe1e14e89b6afc82a3f7a67dc: Time spent starting tablet: real 0.018s user 0.012s sys 0.004s
I20260928 02:15:49.579615 25828 ts_tablet_manager.cc:1428] T 30e8ef1cc9354ebcb66b7dee9fbf91f2 P be6b5f3afc9548b199d7852d0c9fce3a: Time spent starting tablet: real 0.021s user 0.006s sys 0.014s
I20260928 02:15:49.581781 25832 raft_consensus.cc:482] T 30e8ef1cc9354ebcb66b7dee9fbf91f2 P be6b5f3afc9548b199d7852d0c9fce3a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 02:15:49.582118 25832 raft_consensus.cc:504] T 30e8ef1cc9354ebcb66b7dee9fbf91f2 P be6b5f3afc9548b199d7852d0c9fce3a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be6b5f3afc9548b199d7852d0c9fce3a" member_type: VOTER last_known_addr { host: "127.6.235.65" port: 42121 } } peers { permanent_uuid: "4a9e49ebe1e14e89b6afc82a3f7a67dc" member_type: VOTER last_known_addr { host: "127.6.235.66" port: 37759 } } peers { permanent_uuid: "4c8ae0528987411988e503b342f1be5d" member_type: VOTER last_known_addr { host: "127.6.235.67" port: 46293 } }
I20260928 02:15:49.585043 25826 ts_tablet_manager.cc:1428] T 30e8ef1cc9354ebcb66b7dee9fbf91f2 P 4c8ae0528987411988e503b342f1be5d: Time spent starting tablet: real 0.029s user 0.013s sys 0.017s
I20260928 02:15:49.585901 25832 leader_election.cc:288] T 30e8ef1cc9354ebcb66b7dee9fbf91f2 P be6b5f3afc9548b199d7852d0c9fce3a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4a9e49ebe1e14e89b6afc82a3f7a67dc (127.6.235.66:37759), 4c8ae0528987411988e503b342f1be5d (127.6.235.67:46293)
I20260928 02:15:49.586622 25807 heartbeater.cc:503] Master 127.6.235.126:36837 was elected leader, sending a full tablet report...
I20260928 02:15:49.598286 25823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:15:49.587177 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.235.65:48919 (local address 127.6.235.66:37759)
0928 02:15:49.587487 (+ 310us) server_negotiation.cc:206] Beginning negotiation
0928 02:15:49.587497 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 02:15:49.591330 (+ 3833us) server_negotiation.cc:407] Connection header received
0928 02:15:49.591658 (+ 328us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:15:49.591670 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:15:49.591754 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:15:49.591817 (+ 63us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:15:49.592962 (+ 1145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:15:49.594015 (+ 1053us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:15:49.596336 (+ 2321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:15:49.597243 (+ 907us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:15:49.597433 (+ 190us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:15:49.597520 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0928 02:15:49.597850 (+ 330us) server_negotiation.cc:299] Negotiation successful
0928 02:15:49.597989 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":101}
I20260928 02:15:49.598284 25825 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:15:49.588630 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.235.65:32853 (local address 127.6.235.67:46293)
0928 02:15:49.588911 (+ 281us) server_negotiation.cc:206] Beginning negotiation
0928 02:15:49.588919 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 02:15:49.591329 (+ 2410us) server_negotiation.cc:407] Connection header received
0928 02:15:49.591658 (+ 329us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:15:49.591670 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:15:49.591754 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:15:49.591816 (+ 62us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:15:49.593122 (+ 1306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:15:49.594298 (+ 1176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:15:49.596306 (+ 2008us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:15:49.597192 (+ 886us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:15:49.597384 (+ 192us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:15:49.597472 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0928 02:15:49.597839 (+ 367us) server_negotiation.cc:299] Negotiation successful
0928 02:15:49.597996 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260928 02:15:49.598474 25837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:15:49.586044 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.235.66:37759 (local address 127.6.235.65:48919)
0928 02:15:49.591206 (+ 5162us) negotiation.cc:107] Waiting for socket to connect
0928 02:15:49.591232 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0928 02:15:49.591501 (+ 269us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:15:49.592019 (+ 518us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:15:49.592024 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:15:49.592050 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:15:49.592590 (+ 540us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:15:49.592680 (+ 90us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:15:49.594180 (+ 1500us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:15:49.594187 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:15:49.596149 (+ 1962us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:15:49.596164 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:15:49.597487 (+ 1323us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:15:49.597493 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:15:49.597678 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:15:49.597704 (+ 26us) client_negotiation.cc:769] Sending connection context
0928 02:15:49.597808 (+ 104us) client_negotiation.cc:240] Negotiation successful
0928 02:15:49.597987 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4986,"mutex_wait_us":33,"spinlock_wait_cycles":24960,"thread_start_us":5259,"threads_started":1}
I20260928 02:15:49.598513 25836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:15:49.587937 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.235.67:46293 (local address 127.6.235.65:32853)
0928 02:15:49.591237 (+ 3300us) negotiation.cc:107] Waiting for socket to connect
0928 02:15:49.591253 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 02:15:49.591463 (+ 210us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:15:49.592019 (+ 556us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:15:49.592024 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:15:49.592083 (+ 59us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:15:49.592770 (+ 687us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:15:49.592787 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:15:49.594446 (+ 1659us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:15:49.594450 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:15:49.596123 (+ 1673us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:15:49.596140 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:15:49.597439 (+ 1299us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:15:49.597446 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:15:49.597623 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:15:49.597700 (+ 77us) client_negotiation.cc:769] Sending connection context
0928 02:15:49.597814 (+ 114us) client_negotiation.cc:240] Negotiation successful
0928 02:15:49.598004 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3193,"mutex_wait_us":96,"spinlock_wait_cycles":29568,"thread_start_us":2903,"threads_started":1}
I20260928 02:15:49.599689 25628 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "30e8ef1cc9354ebcb66b7dee9fbf91f2" candidate_uuid: "be6b5f3afc9548b199d7852d0c9fce3a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4a9e49ebe1e14e89b6afc82a3f7a67dc" is_pre_election: true
I20260928 02:15:49.599916 25762 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "30e8ef1cc9354ebcb66b7dee9fbf91f2" candidate_uuid: "be6b5f3afc9548b199d7852d0c9fce3a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4c8ae0528987411988e503b342f1be5d" is_pre_election: true
I20260928 02:15:49.600348 25628 raft_consensus.cc:2454] T 30e8ef1cc9354ebcb66b7dee9fbf91f2 P 4a9e49ebe1e14e89b6afc82a3f7a67dc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate be6b5f3afc9548b199d7852d0c9fce3a in term 0.
I20260928 02:15:49.600487 25762 raft_consensus.cc:2454] T 30e8ef1cc9354ebcb66b7dee9fbf91f2 P 4c8ae0528987411988e503b342f1be5d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate be6b5f3afc9548b199d7852d0c9fce3a in term 0.
I20260928 02:15:49.601203 25428 leader_election.cc:302] T 30e8ef1cc9354ebcb66b7dee9fbf91f2 P be6b5f3afc9548b199d7852d0c9fce3a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 4a9e49ebe1e14e89b6afc82a3f7a67dc, 4c8ae0528987411988e503b342f1be5d, be6b5f3afc9548b199d7852d0c9fce3a; no voters:
I20260928 02:15:49.601663 25835 raft_consensus.cc:2781] T 30e8ef1cc9354ebcb66b7dee9fbf91f2 P be6b5f3afc9548b199d7852d0c9fce3a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 02:15:49.601835 25835 raft_consensus.cc:482] T 30e8ef1cc9354ebcb66b7dee9fbf91f2 P be6b5f3afc9548b199d7852d0c9fce3a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 02:15:49.601989 25835 raft_consensus.cc:3037] T 30e8ef1cc9354ebcb66b7dee9fbf91f2 P be6b5f3afc9548b199d7852d0c9fce3a [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:15:49.604353 25835 raft_consensus.cc:504] T 30e8ef1cc9354ebcb66b7dee9fbf91f2 P be6b5f3afc9548b199d7852d0c9fce3a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be6b5f3afc9548b199d7852d0c9fce3a" member_type: VOTER last_known_addr { host: "127.6.235.65" port: 42121 } } peers { permanent_uuid: "4a9e49ebe1e14e89b6afc82a3f7a67dc" member_type: VOTER last_known_addr { host: "127.6.235.66" port: 37759 } } peers { permanent_uuid: "4c8ae0528987411988e503b342f1be5d" member_type: VOTER last_known_addr { host: "127.6.235.67" port: 46293 } }
I20260928 02:15:49.605068 25835 leader_election.cc:288] T 30e8ef1cc9354ebcb66b7dee9fbf91f2 P be6b5f3afc9548b199d7852d0c9fce3a [CANDIDATE]: Term 1 election: Requested vote from peers 4a9e49ebe1e14e89b6afc82a3f7a67dc (127.6.235.66:37759), 4c8ae0528987411988e503b342f1be5d (127.6.235.67:46293)
I20260928 02:15:49.605551 25628 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "30e8ef1cc9354ebcb66b7dee9fbf91f2" candidate_uuid: "be6b5f3afc9548b199d7852d0c9fce3a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4a9e49ebe1e14e89b6afc82a3f7a67dc"
I20260928 02:15:49.605619 25762 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "30e8ef1cc9354ebcb66b7dee9fbf91f2" candidate_uuid: "be6b5f3afc9548b199d7852d0c9fce3a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4c8ae0528987411988e503b342f1be5d"
I20260928 02:15:49.605763 25628 raft_consensus.cc:3037] T 30e8ef1cc9354ebcb66b7dee9fbf91f2 P 4a9e49ebe1e14e89b6afc82a3f7a67dc [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:15:49.605890 25762 raft_consensus.cc:3037] T 30e8ef1cc9354ebcb66b7dee9fbf91f2 P 4c8ae0528987411988e503b342f1be5d [term 0 FOLLOWER]: Advancing to term 1
W20260928 02:15:49.607885 25674 tablet.cc:2369] T 30e8ef1cc9354ebcb66b7dee9fbf91f2 P 4a9e49ebe1e14e89b6afc82a3f7a67dc: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 02:15:49.608191 25628 raft_consensus.cc:2454] T 30e8ef1cc9354ebcb66b7dee9fbf91f2 P 4a9e49ebe1e14e89b6afc82a3f7a67dc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate be6b5f3afc9548b199d7852d0c9fce3a in term 1.
I20260928 02:15:49.608479 25762 raft_consensus.cc:2454] T 30e8ef1cc9354ebcb66b7dee9fbf91f2 P 4c8ae0528987411988e503b342f1be5d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate be6b5f3afc9548b199d7852d0c9fce3a in term 1.
I20260928 02:15:49.608629 25428 leader_election.cc:302] T 30e8ef1cc9354ebcb66b7dee9fbf91f2 P be6b5f3afc9548b199d7852d0c9fce3a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a9e49ebe1e14e89b6afc82a3f7a67dc, be6b5f3afc9548b199d7852d0c9fce3a; no voters:
I20260928 02:15:49.608978 25835 raft_consensus.cc:2781] T 30e8ef1cc9354ebcb66b7dee9fbf91f2 P be6b5f3afc9548b199d7852d0c9fce3a [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:15:49.609211 25835 raft_consensus.cc:686] T 30e8ef1cc9354ebcb66b7dee9fbf91f2 P be6b5f3afc9548b199d7852d0c9fce3a [term 1 LEADER]: Becoming Leader. State: Replica: be6b5f3afc9548b199d7852d0c9fce3a, State: Running, Role: LEADER
I20260928 02:15:49.609659 25835 consensus_queue.cc:237] T 30e8ef1cc9354ebcb66b7dee9fbf91f2 P be6b5f3afc9548b199d7852d0c9fce3a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be6b5f3afc9548b199d7852d0c9fce3a" member_type: VOTER last_known_addr { host: "127.6.235.65" port: 42121 } } peers { permanent_uuid: "4a9e49ebe1e14e89b6afc82a3f7a67dc" member_type: VOTER last_known_addr { host: "127.6.235.66" port: 37759 } } peers { permanent_uuid: "4c8ae0528987411988e503b342f1be5d" member_type: VOTER last_known_addr { host: "127.6.235.67" port: 46293 } }
I20260928 02:15:49.614605 25350 catalog_manager.cc:5495] T 30e8ef1cc9354ebcb66b7dee9fbf91f2 P be6b5f3afc9548b199d7852d0c9fce3a reported cstate change: term changed from 0 to 1, leader changed from <none> to be6b5f3afc9548b199d7852d0c9fce3a (127.6.235.65). New cstate: current_term: 1 leader_uuid: "be6b5f3afc9548b199d7852d0c9fce3a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be6b5f3afc9548b199d7852d0c9fce3a" member_type: VOTER last_known_addr { host: "127.6.235.65" port: 42121 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4a9e49ebe1e14e89b6afc82a3f7a67dc" member_type: VOTER last_known_addr { host: "127.6.235.66" port: 37759 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4c8ae0528987411988e503b342f1be5d" member_type: VOTER last_known_addr { host: "127.6.235.67" port: 46293 } health_report { overall_health: UNKNOWN } } }
I20260928 02:15:49.632699 25822 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:15:49.626683 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58104 (local address 127.6.235.65:42121)
0928 02:15:49.626861 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0928 02:15:49.626866 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 02:15:49.626882 (+ 16us) server_negotiation.cc:407] Connection header received
0928 02:15:49.627064 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:15:49.627068 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:15:49.627152 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:15:49.627285 (+ 133us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 02:15:49.627672 (+ 387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:15:49.628368 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:15:49.630107 (+ 1739us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:15:49.630553 (+ 446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:15:49.630746 (+ 193us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:15:49.631437 (+ 691us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 02:15:49.631986 (+ 549us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 02:15:49.632193 (+ 207us) server_negotiation.cc:1035] Waiting for connection context
0928 02:15:49.632379 (+ 186us) server_negotiation.cc:299] Negotiation successful
0928 02:15:49.632445 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260928 02:15:49.642074 25628 raft_consensus.cc:1261] T 30e8ef1cc9354ebcb66b7dee9fbf91f2 P 4a9e49ebe1e14e89b6afc82a3f7a67dc [term 1 FOLLOWER]: Refusing update from remote peer be6b5f3afc9548b199d7852d0c9fce3a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 02:15:49.642084 25762 raft_consensus.cc:1261] T 30e8ef1cc9354ebcb66b7dee9fbf91f2 P 4c8ae0528987411988e503b342f1be5d [term 1 FOLLOWER]: Refusing update from remote peer be6b5f3afc9548b199d7852d0c9fce3a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 02:15:49.643278 25832 consensus_queue.cc:1035] T 30e8ef1cc9354ebcb66b7dee9fbf91f2 P be6b5f3afc9548b199d7852d0c9fce3a [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a9e49ebe1e14e89b6afc82a3f7a67dc" member_type: VOTER last_known_addr { host: "127.6.235.66" port: 37759 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 02:15:49.643728 25835 consensus_queue.cc:1035] T 30e8ef1cc9354ebcb66b7dee9fbf91f2 P be6b5f3afc9548b199d7852d0c9fce3a [LEADER]: Connected to new peer: Peer: permanent_uuid: "4c8ae0528987411988e503b342f1be5d" member_type: VOTER last_known_addr { host: "127.6.235.67" port: 46293 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 02:15:49.655850 25840 mvcc.cc:205] Tried to move back new op lower bound from 7334140926522376192 to 7334140926404988928. Current Snapshot: MvccSnapshot[applied={T|T < 7334140926522376192}]
I20260928 02:15:49.660296 25842 mvcc.cc:205] Tried to move back new op lower bound from 7334140926522376192 to 7334140926404988928. Current Snapshot: MvccSnapshot[applied={T|T < 7334140926522376192}]
I20260928 02:15:49.676699 25348 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58094:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 02:15:49.677191 25348 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58094:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 02:15:49.682909 25348 catalog_manager.cc:5782] T 00000000000000000000000000000000 P def059cc76c74dfa91d4d5bea5074934: Sending DeleteTablet for 3 replicas of tablet 30e8ef1cc9354ebcb66b7dee9fbf91f2
I20260928 02:15:49.684641 25608 tablet_service.cc:1514] Processing DeleteTablet for tablet 30e8ef1cc9354ebcb66b7dee9fbf91f2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:15:49 UTC) from {username='kudu', principal='kudu/127.6.235.126@KRBTEST.COM'} at 127.0.0.1:52714
I20260928 02:15:49.684800 25742 tablet_service.cc:1514] Processing DeleteTablet for tablet 30e8ef1cc9354ebcb66b7dee9fbf91f2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:15:49 UTC) from {username='kudu', principal='kudu/127.6.235.126@KRBTEST.COM'} at 127.0.0.1:53074
I20260928 02:15:49.685155 25474 tablet_service.cc:1514] Processing DeleteTablet for tablet 30e8ef1cc9354ebcb66b7dee9fbf91f2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:15:49 UTC) from {username='kudu', principal='kudu/127.6.235.126@KRBTEST.COM'} at 127.0.0.1:58100
I20260928 02:15:49.687693 7085 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskTlaUuc/build/asan/bin/kudu pid 25411
I20260928 02:15:49.698755 25851 tablet_replica.cc:330] T 30e8ef1cc9354ebcb66b7dee9fbf91f2 P 4c8ae0528987411988e503b342f1be5d: stopping tablet replica
I20260928 02:15:49.699331 25851 raft_consensus.cc:2229] T 30e8ef1cc9354ebcb66b7dee9fbf91f2 P 4c8ae0528987411988e503b342f1be5d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 02:15:49.699922 25851 raft_consensus.cc:2258] T 30e8ef1cc9354ebcb66b7dee9fbf91f2 P 4c8ae0528987411988e503b342f1be5d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 02:15:49.702708 25851 ts_tablet_manager.cc:1905] T 30e8ef1cc9354ebcb66b7dee9fbf91f2 P 4c8ae0528987411988e503b342f1be5d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 02:15:49.710450 25851 ts_tablet_manager.cc:1918] T 30e8ef1cc9354ebcb66b7dee9fbf91f2 P 4c8ae0528987411988e503b342f1be5d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 02:15:49.710695 25851 log.cc:1196] T 30e8ef1cc9354ebcb66b7dee9fbf91f2 P 4c8ae0528987411988e503b342f1be5d: Deleting WAL directory at /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790561432251112-7085-0/minicluster-data/ts-2/wal/wals/30e8ef1cc9354ebcb66b7dee9fbf91f2
I20260928 02:15:49.711292 25851 ts_tablet_manager.cc:1939] T 30e8ef1cc9354ebcb66b7dee9fbf91f2 P 4c8ae0528987411988e503b342f1be5d: Deleting consensus metadata
I20260928 02:15:49.713487 25336 catalog_manager.cc:4841] TS 4c8ae0528987411988e503b342f1be5d (127.6.235.67:46293): tablet 30e8ef1cc9354ebcb66b7dee9fbf91f2 (table test-table [id=5e3593e2b2a14c9e88f56481f4d8fb84]) successfully deleted
I20260928 02:15:49.714329 25852 tablet_replica.cc:330] T 30e8ef1cc9354ebcb66b7dee9fbf91f2 P be6b5f3afc9548b199d7852d0c9fce3a: stopping tablet replica
I20260928 02:15:49.714860 25852 raft_consensus.cc:2229] T 30e8ef1cc9354ebcb66b7dee9fbf91f2 P be6b5f3afc9548b199d7852d0c9fce3a [term 1 LEADER]: Raft consensus shutting down.
I20260928 02:15:49.715606 25852 raft_consensus.cc:2258] T 30e8ef1cc9354ebcb66b7dee9fbf91f2 P be6b5f3afc9548b199d7852d0c9fce3a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 02:15:49.716470 25850 tablet_replica.cc:330] T 30e8ef1cc9354ebcb66b7dee9fbf91f2 P 4a9e49ebe1e14e89b6afc82a3f7a67dc: stopping tablet replica
I20260928 02:15:49.717000 25850 raft_consensus.cc:2229] T 30e8ef1cc9354ebcb66b7dee9fbf91f2 P 4a9e49ebe1e14e89b6afc82a3f7a67dc [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 02:15:49.717519 25850 raft_consensus.cc:2258] T 30e8ef1cc9354ebcb66b7dee9fbf91f2 P 4a9e49ebe1e14e89b6afc82a3f7a67dc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 02:15:49.718025 25852 ts_tablet_manager.cc:1905] T 30e8ef1cc9354ebcb66b7dee9fbf91f2 P be6b5f3afc9548b199d7852d0c9fce3a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 02:15:49.720739 25850 ts_tablet_manager.cc:1905] T 30e8ef1cc9354ebcb66b7dee9fbf91f2 P 4a9e49ebe1e14e89b6afc82a3f7a67dc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 02:15:49.726584 25852 ts_tablet_manager.cc:1918] T 30e8ef1cc9354ebcb66b7dee9fbf91f2 P be6b5f3afc9548b199d7852d0c9fce3a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 02:15:49.726786 25852 log.cc:1196] T 30e8ef1cc9354ebcb66b7dee9fbf91f2 P be6b5f3afc9548b199d7852d0c9fce3a: Deleting WAL directory at /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790561432251112-7085-0/minicluster-data/ts-0/wal/wals/30e8ef1cc9354ebcb66b7dee9fbf91f2
I20260928 02:15:49.727424 25852 ts_tablet_manager.cc:1939] T 30e8ef1cc9354ebcb66b7dee9fbf91f2 P be6b5f3afc9548b199d7852d0c9fce3a: Deleting consensus metadata
Sep 28 02:15:49 dist-test-slave-86gv krb5kdc[25305](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561747, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.235.65@KRBTEST.COM
I20260928 02:15:49.728821 25850 ts_tablet_manager.cc:1918] T 30e8ef1cc9354ebcb66b7dee9fbf91f2 P 4a9e49ebe1e14e89b6afc82a3f7a67dc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 02:15:49.729054 25850 log.cc:1196] T 30e8ef1cc9354ebcb66b7dee9fbf91f2 P 4a9e49ebe1e14e89b6afc82a3f7a67dc: Deleting WAL directory at /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790561432251112-7085-0/minicluster-data/ts-1/wal/wals/30e8ef1cc9354ebcb66b7dee9fbf91f2
I20260928 02:15:49.729519 25336 catalog_manager.cc:4841] TS be6b5f3afc9548b199d7852d0c9fce3a (127.6.235.65:42121): tablet 30e8ef1cc9354ebcb66b7dee9fbf91f2 (table test-table [id=5e3593e2b2a14c9e88f56481f4d8fb84]) successfully deleted
I20260928 02:15:49.729621 25850 ts_tablet_manager.cc:1939] T 30e8ef1cc9354ebcb66b7dee9fbf91f2 P 4a9e49ebe1e14e89b6afc82a3f7a67dc: Deleting consensus metadata
I20260928 02:15:49.731892 25335 catalog_manager.cc:4841] TS 4a9e49ebe1e14e89b6afc82a3f7a67dc (127.6.235.66:37759): tablet 30e8ef1cc9354ebcb66b7dee9fbf91f2 (table test-table [id=5e3593e2b2a14c9e88f56481f4d8fb84]) successfully deleted
I20260928 02:15:49.736586 25822 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:15:49.690699 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58110 (local address 127.6.235.65:42121)
0928 02:15:49.690976 (+ 277us) server_negotiation.cc:206] Beginning negotiation
0928 02:15:49.690986 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 02:15:49.713002 (+ 22016us) server_negotiation.cc:407] Connection header received
0928 02:15:49.714031 (+ 1029us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:15:49.714040 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:15:49.714175 (+ 135us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:15:49.714277 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:15:49.718858 (+ 4581us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:15:49.720104 (+ 1246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:15:49.722296 (+ 2192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:15:49.723068 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:15:49.723178 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:15:49.729509 (+ 6331us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:15:49.729533 (+ 24us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:15:49.729552 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:15:49.729601 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:15:49.733169 (+ 3568us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:15:49.734240 (+ 1071us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:15:49.734248 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:15:49.734251 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:15:49.734432 (+ 181us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:15:49.734934 (+ 502us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:15:49.734938 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:15:49.734940 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:15:49.735371 (+ 431us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:15:49.735612 (+ 241us) server_negotiation.cc:1035] Waiting for connection context
0928 02:15:49.736014 (+ 402us) server_negotiation.cc:299] Negotiation successful
0928 02:15:49.736256 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":95}
I20260928 02:15:49.737434 25534 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 02:15:49.842929 7085 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTlaUuc/build/asan/bin/kudu with pid 25411
I20260928 02:15:49.863528 7085 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskTlaUuc/build/asan/bin/kudu pid 25545
Sep 28 02:15:49 dist-test-slave-86gv krb5kdc[25305](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561747, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.235.66@KRBTEST.COM
I20260928 02:15:49.882552 25823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:15:49.864573 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52724 (local address 127.6.235.66:37759)
0928 02:15:49.864843 (+ 270us) server_negotiation.cc:206] Beginning negotiation
0928 02:15:49.864848 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 02:15:49.864871 (+ 23us) server_negotiation.cc:407] Connection header received
0928 02:15:49.865125 (+ 254us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:15:49.865130 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:15:49.865235 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:15:49.865340 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:15:49.867190 (+ 1850us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:15:49.868067 (+ 877us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:15:49.870273 (+ 2206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:15:49.870729 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:15:49.870842 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:15:49.876165 (+ 5323us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:15:49.876180 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:15:49.876195 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:15:49.876238 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:15:49.879426 (+ 3188us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:15:49.880263 (+ 837us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:15:49.880271 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:15:49.880274 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:15:49.880401 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:15:49.880835 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:15:49.880838 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:15:49.880840 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:15:49.881345 (+ 505us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:15:49.881626 (+ 281us) server_negotiation.cc:1035] Waiting for connection context
0928 02:15:49.881968 (+ 342us) server_negotiation.cc:299] Negotiation successful
0928 02:15:49.882201 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260928 02:15:49.883265 25668 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 02:15:49.978327 7085 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTlaUuc/build/asan/bin/kudu with pid 25545
I20260928 02:15:49.997807 7085 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskTlaUuc/build/asan/bin/kudu pid 25679
Sep 28 02:15:50 dist-test-slave-86gv krb5kdc[25305](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561747, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.235.67@KRBTEST.COM
I20260928 02:15:50.015436 25825 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:15:49.998710 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53080 (local address 127.6.235.67:46293)
0928 02:15:49.998884 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0928 02:15:49.998889 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 02:15:49.998934 (+ 45us) server_negotiation.cc:407] Connection header received
0928 02:15:49.999229 (+ 295us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:15:49.999234 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:15:49.999318 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:15:49.999441 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:15:50.001046 (+ 1605us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:15:50.001850 (+ 804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:15:50.003315 (+ 1465us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:15:50.003737 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:15:50.003839 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:15:50.008730 (+ 4891us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:15:50.008746 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:15:50.008763 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:15:50.008811 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:15:50.012085 (+ 3274us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:15:50.013124 (+ 1039us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:15:50.013133 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:15:50.013136 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:15:50.013300 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:15:50.013735 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:15:50.013739 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:15:50.013741 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:15:50.014263 (+ 522us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:15:50.014527 (+ 264us) server_negotiation.cc:1035] Waiting for connection context
0928 02:15:50.014879 (+ 352us) server_negotiation.cc:299] Negotiation successful
0928 02:15:50.015167 (+ 288us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260928 02:15:50.016263 25802 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 02:15:50.109572 7085 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTlaUuc/build/asan/bin/kudu with pid 25679
I20260928 02:15:50.128355 7085 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskTlaUuc/build/asan/bin/kudu pid 25318
I20260928 02:15:50.129487 25380 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 02:15:50.246866 7085 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTlaUuc/build/asan/bin/kudu with pid 25318
2026-09-28T02:15:50Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/2 (4494 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/3
2026-09-28T02:15:50Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T02:15:50Z Disabled control of system clock
I20260928 02:15:50.333189 7085 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTlaUuc/build/asan/bin/kudu
/tmp/dist-test-taskTlaUuc/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790561432251112-7085-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790561432251112-7085-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790561432251112-7085-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790561432251112-7085-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.235.126:37673
--webserver_interface=127.6.235.126
--webserver_port=0
--builtin_ntp_servers=127.6.235.84:46397
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.235.126:37673
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260928 02:15:50.702538 25861 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:15:50.702890 25861 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:15:50.703065 25861 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:15:50.710759 25861 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 02:15:50.710901 25861 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:15:50.710983 25861 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:15:50.711076 25861 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 02:15:50.711174 25861 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 02:15:50.721184 25861 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.235.84:46397
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790561432251112-7085-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790561432251112-7085-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.235.126:37673
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.235.126:37673
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790561432251112-7085-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.235.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790561432251112-7085-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 00:51:59 UTC on ccycloud.quasar-axgkwn.root.comops.site
build id 4676464
ASAN enabled
I20260928 02:15:50.722739 25861 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:15:50.724774 25861 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:15:50.738404 25869 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:15:50.738417 25867 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:15:50.738476 25866 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:15:50.739336 25861 server_base.cc:1034] running on GCE node
I20260928 02:15:50.740384 25861 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:15:50.742065 25861 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:15:50.743355 25861 hybrid_clock.cc:648] HybridClock initialized: now 1790561750743284 us; error 69 us; skew 500 ppm
I20260928 02:15:50.743819 25861 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260928 02:15:50.746479 25861 webserver.cc:460] Webserver started at http://127.6.235.126:44457/ using document root <none> and password file <none>
I20260928 02:15:50.747275 25861 fs_manager.cc:359] Metadata directory not provided
I20260928 02:15:50.747416 25861 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:15:50.747768 25861 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:15:50.751037 25861 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790561432251112-7085-0/minicluster-data/master-0/data/instance:
uuid: "9d95ab259c9b4bd6899fd454942615b5"
format_stamp: "Formatted at 2026-09-28 02:15:50 on dist-test-slave-86gv"
server_key: "9cb3f150385454ea427ea5e64562b010"
server_key_iv: "f3754f06ded263778f1ec62d3caa478e"
server_key_version: "encryptionkey@0"
I20260928 02:15:50.751945 25861 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790561432251112-7085-0/minicluster-data/master-0/wal/instance:
uuid: "9d95ab259c9b4bd6899fd454942615b5"
format_stamp: "Formatted at 2026-09-28 02:15:50 on dist-test-slave-86gv"
server_key: "9cb3f150385454ea427ea5e64562b010"
server_key_iv: "f3754f06ded263778f1ec62d3caa478e"
server_key_version: "encryptionkey@0"
I20260928 02:15:50.757897 25861 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.001s
I20260928 02:15:50.762224 25875 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:15:50.763763 25861 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260928 02:15:50.763995 25861 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790561432251112-7085-0/minicluster-data/master-0/data,/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790561432251112-7085-0/minicluster-data/master-0/wal
uuid: "9d95ab259c9b4bd6899fd454942615b5"
format_stamp: "Formatted at 2026-09-28 02:15:50 on dist-test-slave-86gv"
server_key: "9cb3f150385454ea427ea5e64562b010"
server_key_iv: "f3754f06ded263778f1ec62d3caa478e"
server_key_version: "encryptionkey@0"
I20260928 02:15:50.764325 25861 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790561432251112-7085-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790561432251112-7085-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790561432251112-7085-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:15:50.780517 25861 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:15:50.781585 25861 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:15:50.781910 25861 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:15:50.801570 25861 rpc_server.cc:307] RPC server started. Bound to: 127.6.235.126:37673
I20260928 02:15:50.801595 25926 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.235.126:37673 every 8 connection(s)
I20260928 02:15:50.803117 25861 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790561432251112-7085-0/minicluster-data/master-0/data/info.pb
I20260928 02:15:50.807212 25927 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:15:50.811365 7085 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTlaUuc/build/asan/bin/kudu as pid 25861
I20260928 02:15:50.811561 7085 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790561432251112-7085-0/minicluster-data/master-0/wal/instance
I20260928 02:15:50.812139 7085 external_mini_cluster.cc:1428] Setting key b699db7a127e7ec068548fcc6f489a3a
I20260928 02:15:50.821455 25927 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9d95ab259c9b4bd6899fd454942615b5: Bootstrap starting.
I20260928 02:15:50.825417 25927 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9d95ab259c9b4bd6899fd454942615b5: Neither blocks nor log segments found. Creating new log.
I20260928 02:15:50.826452 25927 log.cc:824] T 00000000000000000000000000000000 P 9d95ab259c9b4bd6899fd454942615b5: Log is configured to *not* fsync() on all Append() calls
I20260928 02:15:50.829937 25927 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9d95ab259c9b4bd6899fd454942615b5: No bootstrap required, opened a new log
I20260928 02:15:50.831769 25930 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:15:50.814261 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51796 (local address 127.6.235.126:37673)
0928 02:15:50.815373 (+ 1112us) server_negotiation.cc:206] Beginning negotiation
0928 02:15:50.815385 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0928 02:15:50.815434 (+ 49us) server_negotiation.cc:407] Connection header received
0928 02:15:50.817182 (+ 1748us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:15:50.817207 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:15:50.817761 (+ 554us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:15:50.818148 (+ 387us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:15:50.818772 (+ 624us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:15:50.820296 (+ 1524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:15:50.822195 (+ 1899us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:15:50.822904 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:15:50.823060 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:15:50.828824 (+ 5764us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:15:50.828841 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:15:50.828860 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 02:15:50.829673 (+ 813us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:15:50.829760 (+ 87us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 02:15:50.829790 (+ 30us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:15:50.829893 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0928 02:15:50.830246 (+ 353us) server_negotiation.cc:299] Negotiation successful
0928 02:15:50.830675 (+ 429us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":664,"thread_start_us":519,"threads_started":1}
I20260928 02:15:50.838282 25927 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9d95ab259c9b4bd6899fd454942615b5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9d95ab259c9b4bd6899fd454942615b5" member_type: VOTER last_known_addr { host: "127.6.235.126" port: 37673 } }
I20260928 02:15:50.838711 25927 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9d95ab259c9b4bd6899fd454942615b5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:15:50.838847 25927 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9d95ab259c9b4bd6899fd454942615b5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9d95ab259c9b4bd6899fd454942615b5, State: Initialized, Role: FOLLOWER
I20260928 02:15:50.839555 25927 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9d95ab259c9b4bd6899fd454942615b5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9d95ab259c9b4bd6899fd454942615b5" member_type: VOTER last_known_addr { host: "127.6.235.126" port: 37673 } }
I20260928 02:15:50.839816 25927 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9d95ab259c9b4bd6899fd454942615b5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 02:15:50.839958 25927 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9d95ab259c9b4bd6899fd454942615b5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 02:15:50.840133 25927 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9d95ab259c9b4bd6899fd454942615b5 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:15:50.842779 25927 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9d95ab259c9b4bd6899fd454942615b5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9d95ab259c9b4bd6899fd454942615b5" member_type: VOTER last_known_addr { host: "127.6.235.126" port: 37673 } }
I20260928 02:15:50.843359 25927 leader_election.cc:302] T 00000000000000000000000000000000 P 9d95ab259c9b4bd6899fd454942615b5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9d95ab259c9b4bd6899fd454942615b5; no voters:
I20260928 02:15:50.844019 25927 leader_election.cc:288] T 00000000000000000000000000000000 P 9d95ab259c9b4bd6899fd454942615b5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 02:15:50.844195 25932 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9d95ab259c9b4bd6899fd454942615b5 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:15:50.844998 25932 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9d95ab259c9b4bd6899fd454942615b5 [term 1 LEADER]: Becoming Leader. State: Replica: 9d95ab259c9b4bd6899fd454942615b5, State: Running, Role: LEADER
I20260928 02:15:50.845510 25932 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9d95ab259c9b4bd6899fd454942615b5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9d95ab259c9b4bd6899fd454942615b5" member_type: VOTER last_known_addr { host: "127.6.235.126" port: 37673 } }
I20260928 02:15:50.846282 25927 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9d95ab259c9b4bd6899fd454942615b5 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 02:15:50.848320 25933 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9d95ab259c9b4bd6899fd454942615b5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9d95ab259c9b4bd6899fd454942615b5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9d95ab259c9b4bd6899fd454942615b5" member_type: VOTER last_known_addr { host: "127.6.235.126" port: 37673 } } }
I20260928 02:15:50.848776 25933 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9d95ab259c9b4bd6899fd454942615b5 [sys.catalog]: This master's current role is: LEADER
I20260928 02:15:50.849119 25934 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9d95ab259c9b4bd6899fd454942615b5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9d95ab259c9b4bd6899fd454942615b5. Latest consensus state: current_term: 1 leader_uuid: "9d95ab259c9b4bd6899fd454942615b5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9d95ab259c9b4bd6899fd454942615b5" member_type: VOTER last_known_addr { host: "127.6.235.126" port: 37673 } } }
I20260928 02:15:50.849480 25934 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9d95ab259c9b4bd6899fd454942615b5 [sys.catalog]: This master's current role is: LEADER
I20260928 02:15:50.855268 25937 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 02:15:50.862171 25937 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 02:15:50.872031 25937 catalog_manager.cc:1347] Generated new cluster ID: 53e16739b7ac4078bee309a50251fe92
I20260928 02:15:50.872170 25937 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 02:15:50.888073 25937 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 02:15:50.889664 25937 catalog_manager.cc:1504] Loading token signing keys...
I20260928 02:15:50.899611 25937 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9d95ab259c9b4bd6899fd454942615b5: Generated new TSK 0
I20260928 02:15:50.900743 25937 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 02:15:50.918056 7085 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTlaUuc/build/asan/bin/kudu
/tmp/dist-test-taskTlaUuc/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790561432251112-7085-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790561432251112-7085-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790561432251112-7085-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790561432251112-7085-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.235.65:0
--local_ip_for_outbound_sockets=127.6.235.65
--webserver_interface=127.6.235.65
--webserver_port=0
--tserver_master_addrs=127.6.235.126:37673
--builtin_ntp_servers=127.6.235.84:46397
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260928 02:15:51.307955 25951 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:15:51.308275 25951 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:15:51.308419 25951 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:15:51.316076 25951 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:15:51.316211 25951 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:15:51.316401 25951 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.235.65
I20260928 02:15:51.326620 25951 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.235.84:46397
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790561432251112-7085-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790561432251112-7085-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.235.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790561432251112-7085-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.6.235.65
--webserver_port=0
--tserver_master_addrs=127.6.235.126:37673
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.235.65
--log_dir=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790561432251112-7085-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 00:51:59 UTC on ccycloud.quasar-axgkwn.root.comops.site
build id 4676464
ASAN enabled
I20260928 02:15:51.328217 25951 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:15:51.330148 25951 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:15:51.346060 25957 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:15:51.346138 25956 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:15:51.345916 25959 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:15:51.347209 25951 server_base.cc:1034] running on GCE node
I20260928 02:15:51.348043 25951 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:15:51.349277 25951 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:15:51.350564 25951 hybrid_clock.cc:648] HybridClock initialized: now 1790561751350507 us; error 50 us; skew 500 ppm
I20260928 02:15:51.351042 25951 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260928 02:15:51.353868 25951 webserver.cc:460] Webserver started at http://127.6.235.65:44255/ using document root <none> and password file <none>
I20260928 02:15:51.354990 25951 fs_manager.cc:359] Metadata directory not provided
I20260928 02:15:51.355254 25951 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:15:51.355728 25951 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:15:51.359130 25951 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790561432251112-7085-0/minicluster-data/ts-0/data/instance:
uuid: "f641bde7b5f2435aa4a5f209a0fdd1e4"
format_stamp: "Formatted at 2026-09-28 02:15:51 on dist-test-slave-86gv"
server_key: "0e95e69fe5a6f5069d575c517a8dcad2"
server_key_iv: "c24523961228ed44d4832197cc3c1b2d"
server_key_version: "encryptionkey@0"
I20260928 02:15:51.360103 25951 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790561432251112-7085-0/minicluster-data/ts-0/wal/instance:
uuid: "f641bde7b5f2435aa4a5f209a0fdd1e4"
format_stamp: "Formatted at 2026-09-28 02:15:51 on dist-test-slave-86gv"
server_key: "0e95e69fe5a6f5069d575c517a8dcad2"
server_key_iv: "c24523961228ed44d4832197cc3c1b2d"
server_key_version: "encryptionkey@0"
I20260928 02:15:51.366515 25951 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.008s sys 0.000s
I20260928 02:15:51.371299 25965 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:15:51.373219 25951 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260928 02:15:51.373467 25951 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790561432251112-7085-0/minicluster-data/ts-0/data,/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790561432251112-7085-0/minicluster-data/ts-0/wal
uuid: "f641bde7b5f2435aa4a5f209a0fdd1e4"
format_stamp: "Formatted at 2026-09-28 02:15:51 on dist-test-slave-86gv"
server_key: "0e95e69fe5a6f5069d575c517a8dcad2"
server_key_iv: "c24523961228ed44d4832197cc3c1b2d"
server_key_version: "encryptionkey@0"
I20260928 02:15:51.373773 25951 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790561432251112-7085-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790561432251112-7085-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790561432251112-7085-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:15:51.402992 25951 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:15:51.404060 25951 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:15:51.404439 25951 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:15:51.405701 25951 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:15:51.407577 25951 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:15:51.407729 25951 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:15:51.407871 25951 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:15:51.407994 25951 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:15:51.447494 25951 rpc_server.cc:307] RPC server started. Bound to: 127.6.235.65:34977
I20260928 02:15:51.447549 26077 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.235.65:34977 every 8 connection(s)
I20260928 02:15:51.448938 25951 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790561432251112-7085-0/minicluster-data/ts-0/data/info.pb
I20260928 02:15:51.450798 7085 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTlaUuc/build/asan/bin/kudu as pid 25951
I20260928 02:15:51.451002 7085 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790561432251112-7085-0/minicluster-data/ts-0/wal/instance
I20260928 02:15:51.451603 7085 external_mini_cluster.cc:1428] Setting key 24bfccb5cf8cdf2cb77d767b50a7e0f8
I20260928 02:15:51.455569 7085 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTlaUuc/build/asan/bin/kudu
/tmp/dist-test-taskTlaUuc/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790561432251112-7085-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790561432251112-7085-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790561432251112-7085-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790561432251112-7085-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.235.66:0
--local_ip_for_outbound_sockets=127.6.235.66
--webserver_interface=127.6.235.66
--webserver_port=0
--tserver_master_addrs=127.6.235.126:37673
--builtin_ntp_servers=127.6.235.84:46397
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260928 02:15:51.468525 26080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:15:51.452271 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.235.65:56593 (local address 127.6.235.126:37673)
0928 02:15:51.452927 (+ 656us) server_negotiation.cc:206] Beginning negotiation
0928 02:15:51.452934 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 02:15:51.454729 (+ 1795us) server_negotiation.cc:407] Connection header received
0928 02:15:51.458323 (+ 3594us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:15:51.458328 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:15:51.458441 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:15:51.458559 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:15:51.460421 (+ 1862us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:15:51.461364 (+ 943us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:15:51.464598 (+ 3234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:15:51.465385 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:15:51.465561 (+ 176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:15:51.467286 (+ 1725us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:15:51.467297 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:15:51.467305 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 02:15:51.467339 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:15:51.467398 (+ 59us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 02:15:51.467427 (+ 29us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:15:51.467540 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0928 02:15:51.467802 (+ 262us) server_negotiation.cc:299] Negotiation successful
0928 02:15:51.467943 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":475,"thread_start_us":303,"threads_started":1}
I20260928 02:15:51.469782 26081 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:15:51.453148 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.235.126:37673 (local address 127.6.235.65:56593)
0928 02:15:51.454520 (+ 1372us) negotiation.cc:107] Waiting for socket to connect
0928 02:15:51.454575 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0928 02:15:51.457858 (+ 3283us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:15:51.458917 (+ 1059us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:15:51.458944 (+ 27us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:15:51.459119 (+ 175us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:15:51.460149 (+ 1030us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:15:51.460185 (+ 36us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:15:51.461580 (+ 1395us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:15:51.461590 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:15:51.464348 (+ 2758us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:15:51.464371 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:15:51.465680 (+ 1309us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:15:51.465689 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:15:51.466025 (+ 336us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:15:51.466846 (+ 821us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 02:15:51.466849 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:15:51.466898 (+ 49us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 02:15:51.467008 (+ 110us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:15:51.467556 (+ 548us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:15:51.467567 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:15:51.467574 (+ 7us) client_negotiation.cc:769] Sending connection context
0928 02:15:51.467745 (+ 171us) client_negotiation.cc:240] Negotiation successful
0928 02:15:51.468293 (+ 548us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":593,"thread_start_us":321,"threads_started":1}
I20260928 02:15:51.472091 26078 heartbeater.cc:347] Connected to a master server at 127.6.235.126:37673
I20260928 02:15:51.472546 26078 heartbeater.cc:464] Registering TS with master...
I20260928 02:15:51.473483 26078 heartbeater.cc:511] Master 127.6.235.126:37673 requested a full tablet report, sending...
I20260928 02:15:51.476090 25892 ts_manager.cc:194] Registered new tserver with Master: f641bde7b5f2435aa4a5f209a0fdd1e4 (127.6.235.65:34977)
I20260928 02:15:51.477973 25892 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.6.235.65:56593
W20260928 02:15:51.835896 26082 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:15:51.836242 26082 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:15:51.836388 26082 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:15:51.844409 26082 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:15:51.844556 26082 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:15:51.844781 26082 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.235.66
I20260928 02:15:51.855592 26082 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.235.84:46397
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790561432251112-7085-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790561432251112-7085-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.235.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790561432251112-7085-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.6.235.66
--webserver_port=0
--tserver_master_addrs=127.6.235.126:37673
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.235.66
--log_dir=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790561432251112-7085-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 00:51:59 UTC on ccycloud.quasar-axgkwn.root.comops.site
build id 4676464
ASAN enabled
I20260928 02:15:51.857275 26082 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:15:51.859517 26082 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:15:51.875389 26088 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:15:51.875499 26087 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:15:51.875551 26090 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:15:51.876271 26082 server_base.cc:1034] running on GCE node
I20260928 02:15:51.877079 26082 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:15:51.878225 26082 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:15:51.879473 26082 hybrid_clock.cc:648] HybridClock initialized: now 1790561751879419 us; error 60 us; skew 500 ppm
I20260928 02:15:51.879920 26082 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260928 02:15:51.882871 26082 webserver.cc:460] Webserver started at http://127.6.235.66:36173/ using document root <none> and password file <none>
I20260928 02:15:51.883962 26082 fs_manager.cc:359] Metadata directory not provided
I20260928 02:15:51.884090 26082 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:15:51.884435 26082 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:15:51.887734 26082 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790561432251112-7085-0/minicluster-data/ts-1/data/instance:
uuid: "748deb8124784323865761e33bb15a49"
format_stamp: "Formatted at 2026-09-28 02:15:51 on dist-test-slave-86gv"
server_key: "4109649b8006190021d1eac60e7c6d25"
server_key_iv: "5e1ce4070b8d56fd5c3912c539040809"
server_key_version: "encryptionkey@0"
I20260928 02:15:51.888751 26082 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790561432251112-7085-0/minicluster-data/ts-1/wal/instance:
uuid: "748deb8124784323865761e33bb15a49"
format_stamp: "Formatted at 2026-09-28 02:15:51 on dist-test-slave-86gv"
server_key: "4109649b8006190021d1eac60e7c6d25"
server_key_iv: "5e1ce4070b8d56fd5c3912c539040809"
server_key_version: "encryptionkey@0"
I20260928 02:15:51.894932 26082 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260928 02:15:51.899205 26096 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:15:51.901609 26082 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260928 02:15:51.901831 26082 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790561432251112-7085-0/minicluster-data/ts-1/data,/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790561432251112-7085-0/minicluster-data/ts-1/wal
uuid: "748deb8124784323865761e33bb15a49"
format_stamp: "Formatted at 2026-09-28 02:15:51 on dist-test-slave-86gv"
server_key: "4109649b8006190021d1eac60e7c6d25"
server_key_iv: "5e1ce4070b8d56fd5c3912c539040809"
server_key_version: "encryptionkey@0"
I20260928 02:15:51.902118 26082 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790561432251112-7085-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790561432251112-7085-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790561432251112-7085-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:15:51.921439 26082 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:15:51.922447 26082 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:15:51.922811 26082 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:15:51.923992 26082 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:15:51.925927 26082 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:15:51.926079 26082 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:15:51.926235 26082 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:15:51.926348 26082 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:15:51.964314 26082 rpc_server.cc:307] RPC server started. Bound to: 127.6.235.66:35753
I20260928 02:15:51.964428 26208 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.235.66:35753 every 8 connection(s)
I20260928 02:15:51.965889 26082 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790561432251112-7085-0/minicluster-data/ts-1/data/info.pb
I20260928 02:15:51.970283 7085 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTlaUuc/build/asan/bin/kudu as pid 26082
I20260928 02:15:51.970460 7085 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790561432251112-7085-0/minicluster-data/ts-1/wal/instance
I20260928 02:15:51.970999 7085 external_mini_cluster.cc:1428] Setting key 6b234eb1aa2c332a0bfbc0ec2456470f
I20260928 02:15:51.974073 7085 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTlaUuc/build/asan/bin/kudu
/tmp/dist-test-taskTlaUuc/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790561432251112-7085-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790561432251112-7085-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790561432251112-7085-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790561432251112-7085-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.235.67:0
--local_ip_for_outbound_sockets=127.6.235.67
--webserver_interface=127.6.235.67
--webserver_port=0
--tserver_master_addrs=127.6.235.126:37673
--builtin_ntp_servers=127.6.235.84:46397
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260928 02:15:51.982004 26080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:15:51.968819 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.235.66:39919 (local address 127.6.235.126:37673)
0928 02:15:51.969035 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0928 02:15:51.969040 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 02:15:51.970879 (+ 1839us) server_negotiation.cc:407] Connection header received
0928 02:15:51.973061 (+ 2182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:15:51.973068 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:15:51.973167 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:15:51.973276 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:15:51.974483 (+ 1207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:15:51.975744 (+ 1261us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:15:51.977956 (+ 2212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:15:51.978680 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:15:51.978835 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:15:51.980677 (+ 1842us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:15:51.980687 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:15:51.980695 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 02:15:51.980726 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:15:51.980774 (+ 48us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 02:15:51.980794 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:15:51.980938 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0928 02:15:51.981426 (+ 488us) server_negotiation.cc:299] Negotiation successful
0928 02:15:51.981561 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260928 02:15:51.983139 26211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:15:51.969512 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.235.126:37673 (local address 127.6.235.66:39919)
0928 02:15:51.970672 (+ 1160us) negotiation.cc:107] Waiting for socket to connect
0928 02:15:51.970735 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0928 02:15:51.972613 (+ 1878us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:15:51.973496 (+ 883us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:15:51.973506 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:15:51.973611 (+ 105us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:15:51.974297 (+ 686us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:15:51.974327 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:15:51.975969 (+ 1642us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:15:51.975974 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:15:51.977720 (+ 1746us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:15:51.977736 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:15:51.979012 (+ 1276us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:15:51.979022 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:15:51.979341 (+ 319us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:15:51.980258 (+ 917us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 02:15:51.980261 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:15:51.980311 (+ 50us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 02:15:51.980412 (+ 101us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:15:51.981146 (+ 734us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:15:51.981157 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:15:51.981165 (+ 8us) client_negotiation.cc:769] Sending connection context
0928 02:15:51.981389 (+ 224us) client_negotiation.cc:240] Negotiation successful
0928 02:15:51.981782 (+ 393us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":533,"thread_start_us":323,"threads_started":1}
I20260928 02:15:51.984944 26209 heartbeater.cc:347] Connected to a master server at 127.6.235.126:37673
I20260928 02:15:51.985311 26209 heartbeater.cc:464] Registering TS with master...
I20260928 02:15:51.986218 26209 heartbeater.cc:511] Master 127.6.235.126:37673 requested a full tablet report, sending...
I20260928 02:15:51.987987 25892 ts_manager.cc:194] Registered new tserver with Master: 748deb8124784323865761e33bb15a49 (127.6.235.66:35753)
I20260928 02:15:51.989039 25892 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.6.235.66:39919
W20260928 02:15:52.329242 26212 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:15:52.329612 26212 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:15:52.329777 26212 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:15:52.337646 26212 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:15:52.337792 26212 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:15:52.337965 26212 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.235.67
I20260928 02:15:52.347880 26212 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.235.84:46397
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790561432251112-7085-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790561432251112-7085-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.235.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790561432251112-7085-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.6.235.67
--webserver_port=0
--tserver_master_addrs=127.6.235.126:37673
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.235.67
--log_dir=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790561432251112-7085-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 00:51:59 UTC on ccycloud.quasar-axgkwn.root.comops.site
build id 4676464
ASAN enabled
I20260928 02:15:52.349375 26212 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:15:52.351264 26212 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:15:52.365712 26217 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:15:52.365677 26218 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:15:52.366532 26220 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:15:52.367449 26212 server_base.cc:1034] running on GCE node
I20260928 02:15:52.368192 26212 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:15:52.369400 26212 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:15:52.370636 26212 hybrid_clock.cc:648] HybridClock initialized: now 1790561752370597 us; error 30 us; skew 500 ppm
I20260928 02:15:52.371013 26212 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260928 02:15:52.374269 26212 webserver.cc:460] Webserver started at http://127.6.235.67:40295/ using document root <none> and password file <none>
I20260928 02:15:52.375108 26212 fs_manager.cc:359] Metadata directory not provided
I20260928 02:15:52.375214 26212 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:15:52.375635 26212 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:15:52.379118 26212 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790561432251112-7085-0/minicluster-data/ts-2/data/instance:
uuid: "238723f3f6594ba69f78f47084af7896"
format_stamp: "Formatted at 2026-09-28 02:15:52 on dist-test-slave-86gv"
server_key: "0a1ce449267d2dd9e7b286ca1f738e81"
server_key_iv: "acc1cd9e98ab2ff408eb726ca1cbc381"
server_key_version: "encryptionkey@0"
I20260928 02:15:52.380143 26212 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790561432251112-7085-0/minicluster-data/ts-2/wal/instance:
uuid: "238723f3f6594ba69f78f47084af7896"
format_stamp: "Formatted at 2026-09-28 02:15:52 on dist-test-slave-86gv"
server_key: "0a1ce449267d2dd9e7b286ca1f738e81"
server_key_iv: "acc1cd9e98ab2ff408eb726ca1cbc381"
server_key_version: "encryptionkey@0"
I20260928 02:15:52.386179 26212 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.003s sys 0.005s
I20260928 02:15:52.390444 26226 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:15:52.392285 26212 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260928 02:15:52.392524 26212 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790561432251112-7085-0/minicluster-data/ts-2/data,/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790561432251112-7085-0/minicluster-data/ts-2/wal
uuid: "238723f3f6594ba69f78f47084af7896"
format_stamp: "Formatted at 2026-09-28 02:15:52 on dist-test-slave-86gv"
server_key: "0a1ce449267d2dd9e7b286ca1f738e81"
server_key_iv: "acc1cd9e98ab2ff408eb726ca1cbc381"
server_key_version: "encryptionkey@0"
I20260928 02:15:52.392802 26212 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790561432251112-7085-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790561432251112-7085-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790561432251112-7085-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:15:52.412405 26212 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:15:52.413424 26212 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:15:52.413775 26212 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:15:52.414867 26212 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:15:52.416478 26212 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:15:52.416611 26212 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:15:52.416751 26212 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:15:52.416841 26212 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:15:52.453823 26212 rpc_server.cc:307] RPC server started. Bound to: 127.6.235.67:34931
I20260928 02:15:52.453863 26338 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.235.67:34931 every 8 connection(s)
I20260928 02:15:52.455441 26212 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790561432251112-7085-0/minicluster-data/ts-2/data/info.pb
I20260928 02:15:52.464418 7085 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTlaUuc/build/asan/bin/kudu as pid 26212
I20260928 02:15:52.464619 7085 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790561432251112-7085-0/minicluster-data/ts-2/wal/instance
I20260928 02:15:52.465134 7085 external_mini_cluster.cc:1428] Setting key 2036ce630c5707f3cd98ace03559a4ab
I20260928 02:15:52.471120 26080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:15:52.458578 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.235.67:37673 (local address 127.6.235.126:37673)
0928 02:15:52.458816 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0928 02:15:52.458823 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 02:15:52.460805 (+ 1982us) server_negotiation.cc:407] Connection header received
0928 02:15:52.462881 (+ 2076us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:15:52.462886 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:15:52.462970 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:15:52.463076 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:15:52.464684 (+ 1608us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:15:52.465632 (+ 948us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:15:52.467398 (+ 1766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:15:52.467997 (+ 599us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:15:52.468140 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:15:52.470149 (+ 2009us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:15:52.470163 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:15:52.470166 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 02:15:52.470189 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:15:52.470224 (+ 35us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 02:15:52.470239 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:15:52.470352 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0928 02:15:52.470634 (+ 282us) server_negotiation.cc:299] Negotiation successful
0928 02:15:52.470761 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":88}
I20260928 02:15:52.472290 26341 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:15:52.459380 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.235.126:37673 (local address 127.6.235.67:37673)
0928 02:15:52.460591 (+ 1211us) negotiation.cc:107] Waiting for socket to connect
0928 02:15:52.460635 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0928 02:15:52.462531 (+ 1896us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:15:52.463406 (+ 875us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:15:52.463416 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:15:52.463530 (+ 114us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:15:52.464415 (+ 885us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:15:52.464446 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:15:52.465813 (+ 1367us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:15:52.465823 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:15:52.467222 (+ 1399us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:15:52.467237 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:15:52.468366 (+ 1129us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:15:52.468377 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:15:52.468637 (+ 260us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:15:52.469655 (+ 1018us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 02:15:52.469659 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:15:52.469708 (+ 49us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 02:15:52.469811 (+ 103us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:15:52.470359 (+ 548us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:15:52.470375 (+ 16us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:15:52.470385 (+ 10us) client_negotiation.cc:769] Sending connection context
0928 02:15:52.470618 (+ 233us) client_negotiation.cc:240] Negotiation successful
0928 02:15:52.471140 (+ 522us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":586,"thread_start_us":348,"threads_started":1}
I20260928 02:15:52.474272 26339 heartbeater.cc:347] Connected to a master server at 127.6.235.126:37673
I20260928 02:15:52.474694 26339 heartbeater.cc:464] Registering TS with master...
I20260928 02:15:52.475602 26339 heartbeater.cc:511] Master 127.6.235.126:37673 requested a full tablet report, sending...
I20260928 02:15:52.477514 25892 ts_manager.cc:194] Registered new tserver with Master: 238723f3f6594ba69f78f47084af7896 (127.6.235.67:34931)
I20260928 02:15:52.478554 25892 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.6.235.67:37673
I20260928 02:15:52.481341 7085 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 02:15:52.481748 26078 heartbeater.cc:503] Master 127.6.235.126:37673 was elected leader, sending a full tablet report...
I20260928 02:15:52.494138 26080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:15:52.486983 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41828 (local address 127.6.235.126:37673)
0928 02:15:52.487221 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0928 02:15:52.487226 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 02:15:52.487593 (+ 367us) server_negotiation.cc:407] Connection header received
0928 02:15:52.487821 (+ 228us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:15:52.487825 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:15:52.487901 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:15:52.488086 (+ 185us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:15:52.488639 (+ 553us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:15:52.489547 (+ 908us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:15:52.491315 (+ 1768us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:15:52.491753 (+ 438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:15:52.491867 (+ 114us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:15:52.493300 (+ 1433us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:15:52.493305 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:15:52.493309 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 02:15:52.493337 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:15:52.493389 (+ 52us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 02:15:52.493403 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:15:52.493548 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0928 02:15:52.493805 (+ 257us) server_negotiation.cc:299] Negotiation successful
0928 02:15:52.493862 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260928 02:15:52.499593 25892 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41828:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 02:15:52.507093 25892 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 02:15:52.532653 26352 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:15:52.522486 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48032 (local address 127.6.235.67:34931)
0928 02:15:52.524479 (+ 1993us) server_negotiation.cc:206] Beginning negotiation
0928 02:15:52.524490 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 02:15:52.524533 (+ 43us) server_negotiation.cc:407] Connection header received
0928 02:15:52.524643 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:15:52.524651 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:15:52.525002 (+ 351us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:15:52.525228 (+ 226us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:15:52.525945 (+ 717us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:15:52.527677 (+ 1732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:15:52.530229 (+ 2552us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:15:52.531456 (+ 1227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:15:52.531620 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:15:52.531924 (+ 304us) server_negotiation.cc:1035] Waiting for connection context
0928 02:15:52.532084 (+ 160us) server_negotiation.cc:299] Negotiation successful
0928 02:15:52.532262 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1809,"thread_start_us":1662,"threads_started":1}
I20260928 02:15:52.533054 26351 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:15:52.522321 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.235.67:34931 (local address 127.0.0.1:48032)
0928 02:15:52.523175 (+ 854us) negotiation.cc:107] Waiting for socket to connect
0928 02:15:52.523222 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0928 02:15:52.523398 (+ 176us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:15:52.525235 (+ 1837us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:15:52.525243 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:15:52.525279 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:15:52.525720 (+ 441us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:15:52.525736 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:15:52.527868 (+ 2132us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:15:52.527873 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:15:52.529986 (+ 2113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:15:52.529998 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:15:52.531652 (+ 1654us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:15:52.531659 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:15:52.531857 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:15:52.531906 (+ 49us) client_negotiation.cc:769] Sending connection context
0928 02:15:52.532033 (+ 127us) client_negotiation.cc:240] Negotiation successful
0928 02:15:52.532247 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":540,"mutex_wait_us":26,"spinlock_wait_cycles":13184,"thread_start_us":389,"threads_started":1}
I20260928 02:15:52.534185 26353 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:15:52.523649 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.235.66:35753 (local address 127.0.0.1:37764)
0928 02:15:52.525193 (+ 1544us) negotiation.cc:107] Waiting for socket to connect
0928 02:15:52.525213 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0928 02:15:52.525336 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:15:52.526494 (+ 1158us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:15:52.526503 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:15:52.526578 (+ 75us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:15:52.527072 (+ 494us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:15:52.527087 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:15:52.529094 (+ 2007us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:15:52.529101 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:15:52.531287 (+ 2186us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:15:52.531302 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:15:52.533266 (+ 1964us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:15:52.533273 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:15:52.533478 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:15:52.533518 (+ 40us) client_negotiation.cc:769] Sending connection context
0928 02:15:52.533642 (+ 124us) client_negotiation.cc:240] Negotiation successful
0928 02:15:52.533780 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1405,"mutex_wait_us":24,"thread_start_us":1251,"threads_started":1}
I20260928 02:15:52.534919 26354 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:15:52.523978 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37764 (local address 127.6.235.66:35753)
0928 02:15:52.524762 (+ 784us) server_negotiation.cc:206] Beginning negotiation
0928 02:15:52.524769 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 02:15:52.525482 (+ 713us) server_negotiation.cc:407] Connection header received
0928 02:15:52.525603 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:15:52.525611 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:15:52.525874 (+ 263us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:15:52.526063 (+ 189us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:15:52.527299 (+ 1236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:15:52.528948 (+ 1649us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:15:52.531507 (+ 2559us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:15:52.533022 (+ 1515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:15:52.533200 (+ 178us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:15:52.533473 (+ 273us) server_negotiation.cc:1035] Waiting for connection context
0928 02:15:52.533692 (+ 219us) server_negotiation.cc:299] Negotiation successful
0928 02:15:52.533855 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":625,"thread_start_us":494,"threads_started":1}
I20260928 02:15:52.537552 26356 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:15:52.525988 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.235.65:34977 (local address 127.0.0.1:54374)
0928 02:15:52.527922 (+ 1934us) negotiation.cc:107] Waiting for socket to connect
0928 02:15:52.527946 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0928 02:15:52.528094 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:15:52.528690 (+ 596us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:15:52.528694 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:15:52.528719 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:15:52.529296 (+ 577us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:15:52.529315 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:15:52.532247 (+ 2932us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:15:52.532252 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:15:52.534409 (+ 2157us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:15:52.534424 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:15:52.536728 (+ 2304us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:15:52.536735 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:15:52.536959 (+ 224us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:15:52.536992 (+ 33us) client_negotiation.cc:769] Sending connection context
0928 02:15:52.537092 (+ 100us) client_negotiation.cc:240] Negotiation successful
0928 02:15:52.537231 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1797,"thread_start_us":1649,"threads_started":1}
I20260928 02:15:52.537984 26355 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:15:52.525662 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54374 (local address 127.6.235.65:34977)
0928 02:15:52.527470 (+ 1808us) server_negotiation.cc:206] Beginning negotiation
0928 02:15:52.527479 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 02:15:52.528050 (+ 571us) server_negotiation.cc:407] Connection header received
0928 02:15:52.528289 (+ 239us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:15:52.528295 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:15:52.528494 (+ 199us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:15:52.528707 (+ 213us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:15:52.529498 (+ 791us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:15:52.532076 (+ 2578us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:15:52.534656 (+ 2580us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:15:52.536505 (+ 1849us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:15:52.536656 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:15:52.536951 (+ 295us) server_negotiation.cc:1035] Waiting for connection context
0928 02:15:52.537165 (+ 214us) server_negotiation.cc:299] Negotiation successful
0928 02:15:52.537627 (+ 462us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1639,"thread_start_us":1519,"threads_started":1}
I20260928 02:15:52.537803 26274 tablet_service.cc:1467] Processing CreateTablet for tablet f8cb44ccf21f41a3b15bfad27cac0fee (DEFAULT_TABLE table=test-table [id=4550152b62134a72913b2fb65ef23d98]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:15:52.538709 26144 tablet_service.cc:1467] Processing CreateTablet for tablet f8cb44ccf21f41a3b15bfad27cac0fee (DEFAULT_TABLE table=test-table [id=4550152b62134a72913b2fb65ef23d98]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:15:52.539482 26274 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f8cb44ccf21f41a3b15bfad27cac0fee. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:15:52.541977 26144 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f8cb44ccf21f41a3b15bfad27cac0fee. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:15:52.543295 26013 tablet_service.cc:1467] Processing CreateTablet for tablet f8cb44ccf21f41a3b15bfad27cac0fee (DEFAULT_TABLE table=test-table [id=4550152b62134a72913b2fb65ef23d98]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:15:52.544919 26013 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f8cb44ccf21f41a3b15bfad27cac0fee. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:15:52.555159 26357 tablet_bootstrap.cc:492] T f8cb44ccf21f41a3b15bfad27cac0fee P 748deb8124784323865761e33bb15a49: Bootstrap starting.
I20260928 02:15:52.558862 26357 tablet_bootstrap.cc:654] T f8cb44ccf21f41a3b15bfad27cac0fee P 748deb8124784323865761e33bb15a49: Neither blocks nor log segments found. Creating new log.
I20260928 02:15:52.560371 26357 log.cc:824] T f8cb44ccf21f41a3b15bfad27cac0fee P 748deb8124784323865761e33bb15a49: Log is configured to *not* fsync() on all Append() calls
I20260928 02:15:52.560426 26358 tablet_bootstrap.cc:492] T f8cb44ccf21f41a3b15bfad27cac0fee P 238723f3f6594ba69f78f47084af7896: Bootstrap starting.
I20260928 02:15:52.562410 26360 tablet_bootstrap.cc:492] T f8cb44ccf21f41a3b15bfad27cac0fee P f641bde7b5f2435aa4a5f209a0fdd1e4: Bootstrap starting.
I20260928 02:15:52.564373 26357 tablet_bootstrap.cc:492] T f8cb44ccf21f41a3b15bfad27cac0fee P 748deb8124784323865761e33bb15a49: No bootstrap required, opened a new log
I20260928 02:15:52.564689 26357 ts_tablet_manager.cc:1397] T f8cb44ccf21f41a3b15bfad27cac0fee P 748deb8124784323865761e33bb15a49: Time spent bootstrapping tablet: real 0.010s user 0.004s sys 0.004s
I20260928 02:15:52.565187 26358 tablet_bootstrap.cc:654] T f8cb44ccf21f41a3b15bfad27cac0fee P 238723f3f6594ba69f78f47084af7896: Neither blocks nor log segments found. Creating new log.
I20260928 02:15:52.566083 26360 tablet_bootstrap.cc:654] T f8cb44ccf21f41a3b15bfad27cac0fee P f641bde7b5f2435aa4a5f209a0fdd1e4: Neither blocks nor log segments found. Creating new log.
I20260928 02:15:52.566500 26358 log.cc:824] T f8cb44ccf21f41a3b15bfad27cac0fee P 238723f3f6594ba69f78f47084af7896: Log is configured to *not* fsync() on all Append() calls
I20260928 02:15:52.567356 26360 log.cc:824] T f8cb44ccf21f41a3b15bfad27cac0fee P f641bde7b5f2435aa4a5f209a0fdd1e4: Log is configured to *not* fsync() on all Append() calls
I20260928 02:15:52.570480 26358 tablet_bootstrap.cc:492] T f8cb44ccf21f41a3b15bfad27cac0fee P 238723f3f6594ba69f78f47084af7896: No bootstrap required, opened a new log
I20260928 02:15:52.570905 26358 ts_tablet_manager.cc:1397] T f8cb44ccf21f41a3b15bfad27cac0fee P 238723f3f6594ba69f78f47084af7896: Time spent bootstrapping tablet: real 0.011s user 0.008s sys 0.000s
I20260928 02:15:52.573093 26357 raft_consensus.cc:348] T f8cb44ccf21f41a3b15bfad27cac0fee P 748deb8124784323865761e33bb15a49 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "238723f3f6594ba69f78f47084af7896" member_type: VOTER last_known_addr { host: "127.6.235.67" port: 34931 } } peers { permanent_uuid: "748deb8124784323865761e33bb15a49" member_type: VOTER last_known_addr { host: "127.6.235.66" port: 35753 } } peers { permanent_uuid: "f641bde7b5f2435aa4a5f209a0fdd1e4" member_type: VOTER last_known_addr { host: "127.6.235.65" port: 34977 } }
I20260928 02:15:52.573661 26357 raft_consensus.cc:374] T f8cb44ccf21f41a3b15bfad27cac0fee P 748deb8124784323865761e33bb15a49 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:15:52.573769 26357 raft_consensus.cc:729] T f8cb44ccf21f41a3b15bfad27cac0fee P 748deb8124784323865761e33bb15a49 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 748deb8124784323865761e33bb15a49, State: Initialized, Role: FOLLOWER
I20260928 02:15:52.574496 26357 consensus_queue.cc:260] T f8cb44ccf21f41a3b15bfad27cac0fee P 748deb8124784323865761e33bb15a49 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "238723f3f6594ba69f78f47084af7896" member_type: VOTER last_known_addr { host: "127.6.235.67" port: 34931 } } peers { permanent_uuid: "748deb8124784323865761e33bb15a49" member_type: VOTER last_known_addr { host: "127.6.235.66" port: 35753 } } peers { permanent_uuid: "f641bde7b5f2435aa4a5f209a0fdd1e4" member_type: VOTER last_known_addr { host: "127.6.235.65" port: 34977 } }
I20260928 02:15:52.576256 26360 tablet_bootstrap.cc:492] T f8cb44ccf21f41a3b15bfad27cac0fee P f641bde7b5f2435aa4a5f209a0fdd1e4: No bootstrap required, opened a new log
I20260928 02:15:52.576581 26360 ts_tablet_manager.cc:1397] T f8cb44ccf21f41a3b15bfad27cac0fee P f641bde7b5f2435aa4a5f209a0fdd1e4: Time spent bootstrapping tablet: real 0.014s user 0.004s sys 0.008s
I20260928 02:15:52.582710 26358 raft_consensus.cc:348] T f8cb44ccf21f41a3b15bfad27cac0fee P 238723f3f6594ba69f78f47084af7896 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "238723f3f6594ba69f78f47084af7896" member_type: VOTER last_known_addr { host: "127.6.235.67" port: 34931 } } peers { permanent_uuid: "748deb8124784323865761e33bb15a49" member_type: VOTER last_known_addr { host: "127.6.235.66" port: 35753 } } peers { permanent_uuid: "f641bde7b5f2435aa4a5f209a0fdd1e4" member_type: VOTER last_known_addr { host: "127.6.235.65" port: 34977 } }
I20260928 02:15:52.583359 26358 raft_consensus.cc:374] T f8cb44ccf21f41a3b15bfad27cac0fee P 238723f3f6594ba69f78f47084af7896 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:15:52.583472 26358 raft_consensus.cc:729] T f8cb44ccf21f41a3b15bfad27cac0fee P 238723f3f6594ba69f78f47084af7896 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 238723f3f6594ba69f78f47084af7896, State: Initialized, Role: FOLLOWER
I20260928 02:15:52.584388 26209 heartbeater.cc:503] Master 127.6.235.126:37673 was elected leader, sending a full tablet report...
I20260928 02:15:52.584331 26358 consensus_queue.cc:260] T f8cb44ccf21f41a3b15bfad27cac0fee P 238723f3f6594ba69f78f47084af7896 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "238723f3f6594ba69f78f47084af7896" member_type: VOTER last_known_addr { host: "127.6.235.67" port: 34931 } } peers { permanent_uuid: "748deb8124784323865761e33bb15a49" member_type: VOTER last_known_addr { host: "127.6.235.66" port: 35753 } } peers { permanent_uuid: "f641bde7b5f2435aa4a5f209a0fdd1e4" member_type: VOTER last_known_addr { host: "127.6.235.65" port: 34977 } }
I20260928 02:15:52.585209 26357 ts_tablet_manager.cc:1428] T f8cb44ccf21f41a3b15bfad27cac0fee P 748deb8124784323865761e33bb15a49: Time spent starting tablet: real 0.020s user 0.013s sys 0.008s
I20260928 02:15:52.586089 26339 heartbeater.cc:503] Master 127.6.235.126:37673 was elected leader, sending a full tablet report...
I20260928 02:15:52.586223 26358 ts_tablet_manager.cc:1428] T f8cb44ccf21f41a3b15bfad27cac0fee P 238723f3f6594ba69f78f47084af7896: Time spent starting tablet: real 0.015s user 0.007s sys 0.009s
I20260928 02:15:52.590409 26360 raft_consensus.cc:348] T f8cb44ccf21f41a3b15bfad27cac0fee P f641bde7b5f2435aa4a5f209a0fdd1e4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "238723f3f6594ba69f78f47084af7896" member_type: VOTER last_known_addr { host: "127.6.235.67" port: 34931 } } peers { permanent_uuid: "748deb8124784323865761e33bb15a49" member_type: VOTER last_known_addr { host: "127.6.235.66" port: 35753 } } peers { permanent_uuid: "f641bde7b5f2435aa4a5f209a0fdd1e4" member_type: VOTER last_known_addr { host: "127.6.235.65" port: 34977 } }
I20260928 02:15:52.590965 26360 raft_consensus.cc:374] T f8cb44ccf21f41a3b15bfad27cac0fee P f641bde7b5f2435aa4a5f209a0fdd1e4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:15:52.591076 26360 raft_consensus.cc:729] T f8cb44ccf21f41a3b15bfad27cac0fee P f641bde7b5f2435aa4a5f209a0fdd1e4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f641bde7b5f2435aa4a5f209a0fdd1e4, State: Initialized, Role: FOLLOWER
I20260928 02:15:52.591814 26360 consensus_queue.cc:260] T f8cb44ccf21f41a3b15bfad27cac0fee P f641bde7b5f2435aa4a5f209a0fdd1e4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "238723f3f6594ba69f78f47084af7896" member_type: VOTER last_known_addr { host: "127.6.235.67" port: 34931 } } peers { permanent_uuid: "748deb8124784323865761e33bb15a49" member_type: VOTER last_known_addr { host: "127.6.235.66" port: 35753 } } peers { permanent_uuid: "f641bde7b5f2435aa4a5f209a0fdd1e4" member_type: VOTER last_known_addr { host: "127.6.235.65" port: 34977 } }
I20260928 02:15:52.593595 26360 ts_tablet_manager.cc:1428] T f8cb44ccf21f41a3b15bfad27cac0fee P f641bde7b5f2435aa4a5f209a0fdd1e4: Time spent starting tablet: real 0.017s user 0.010s sys 0.005s
W20260928 02:15:52.701756 26079 tablet.cc:2369] T f8cb44ccf21f41a3b15bfad27cac0fee P f641bde7b5f2435aa4a5f209a0fdd1e4: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 02:15:52.707530 26340 tablet.cc:2369] T f8cb44ccf21f41a3b15bfad27cac0fee P 238723f3f6594ba69f78f47084af7896: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 02:15:52.718360 26210 tablet.cc:2369] T f8cb44ccf21f41a3b15bfad27cac0fee P 748deb8124784323865761e33bb15a49: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 02:15:52.733213 26363 raft_consensus.cc:482] T f8cb44ccf21f41a3b15bfad27cac0fee P 748deb8124784323865761e33bb15a49 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 02:15:52.733568 26363 raft_consensus.cc:504] T f8cb44ccf21f41a3b15bfad27cac0fee P 748deb8124784323865761e33bb15a49 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "238723f3f6594ba69f78f47084af7896" member_type: VOTER last_known_addr { host: "127.6.235.67" port: 34931 } } peers { permanent_uuid: "748deb8124784323865761e33bb15a49" member_type: VOTER last_known_addr { host: "127.6.235.66" port: 35753 } } peers { permanent_uuid: "f641bde7b5f2435aa4a5f209a0fdd1e4" member_type: VOTER last_known_addr { host: "127.6.235.65" port: 34977 } }
I20260928 02:15:52.735317 26363 leader_election.cc:288] T f8cb44ccf21f41a3b15bfad27cac0fee P 748deb8124784323865761e33bb15a49 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 238723f3f6594ba69f78f47084af7896 (127.6.235.67:34931), f641bde7b5f2435aa4a5f209a0fdd1e4 (127.6.235.65:34977)
I20260928 02:15:52.742817 26352 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:15:52.735889 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.235.66:41347 (local address 127.6.235.67:34931)
0928 02:15:52.736179 (+ 290us) server_negotiation.cc:206] Beginning negotiation
0928 02:15:52.736185 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 02:15:52.736786 (+ 601us) server_negotiation.cc:407] Connection header received
0928 02:15:52.737059 (+ 273us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:15:52.737069 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:15:52.737164 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:15:52.737226 (+ 62us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:15:52.737952 (+ 726us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:15:52.738695 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:15:52.740763 (+ 2068us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:15:52.741862 (+ 1099us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:15:52.742078 (+ 216us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:15:52.742159 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0928 02:15:52.742428 (+ 269us) server_negotiation.cc:299] Negotiation successful
0928 02:15:52.742555 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260928 02:15:52.742980 26366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:15:52.735889 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.235.67:34931 (local address 127.6.235.66:41347)
0928 02:15:52.736647 (+ 758us) negotiation.cc:107] Waiting for socket to connect
0928 02:15:52.736670 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0928 02:15:52.736819 (+ 149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:15:52.737414 (+ 595us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:15:52.737420 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:15:52.737443 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:15:52.737801 (+ 358us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:15:52.737809 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:15:52.738832 (+ 1023us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:15:52.738840 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:15:52.740517 (+ 1677us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:15:52.740530 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:15:52.742074 (+ 1544us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:15:52.742081 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:15:52.742257 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:15:52.742284 (+ 27us) client_negotiation.cc:769] Sending connection context
0928 02:15:52.742401 (+ 117us) client_negotiation.cc:240] Negotiation successful
0928 02:15:52.742555 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":620,"mutex_wait_us":5,"thread_start_us":422,"threads_started":1}
I20260928 02:15:52.743516 26355 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:15:52.737051 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.235.66:49271 (local address 127.6.235.65:34977)
0928 02:15:52.737338 (+ 287us) server_negotiation.cc:206] Beginning negotiation
0928 02:15:52.737344 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 02:15:52.737557 (+ 213us) server_negotiation.cc:407] Connection header received
0928 02:15:52.737814 (+ 257us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:15:52.737821 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:15:52.737897 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:15:52.737997 (+ 100us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:15:52.738558 (+ 561us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:15:52.739244 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:15:52.741547 (+ 2303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:15:52.742455 (+ 908us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:15:52.742681 (+ 226us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:15:52.742759 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0928 02:15:52.743133 (+ 374us) server_negotiation.cc:299] Negotiation successful
0928 02:15:52.743301 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":90}
I20260928 02:15:52.744280 26367 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:15:52.736787 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.235.65:34977 (local address 127.6.235.66:49271)
0928 02:15:52.737434 (+ 647us) negotiation.cc:107] Waiting for socket to connect
0928 02:15:52.737449 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 02:15:52.737661 (+ 212us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:15:52.738011 (+ 350us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:15:52.738016 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:15:52.738039 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:15:52.738361 (+ 322us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:15:52.738380 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:15:52.739388 (+ 1008us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:15:52.739393 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:15:52.741323 (+ 1930us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:15:52.741337 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:15:52.742682 (+ 1345us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:15:52.742689 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:15:52.742878 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:15:52.742953 (+ 75us) client_negotiation.cc:769] Sending connection context
0928 02:15:52.743762 (+ 809us) client_negotiation.cc:240] Negotiation successful
0928 02:15:52.743940 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":512,"thread_start_us":304,"threads_started":1}
I20260928 02:15:52.744177 26294 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f8cb44ccf21f41a3b15bfad27cac0fee" candidate_uuid: "748deb8124784323865761e33bb15a49" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "238723f3f6594ba69f78f47084af7896" is_pre_election: true
I20260928 02:15:52.744962 26294 raft_consensus.cc:2454] T f8cb44ccf21f41a3b15bfad27cac0fee P 238723f3f6594ba69f78f47084af7896 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 748deb8124784323865761e33bb15a49 in term 0.
I20260928 02:15:52.745471 26033 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f8cb44ccf21f41a3b15bfad27cac0fee" candidate_uuid: "748deb8124784323865761e33bb15a49" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f641bde7b5f2435aa4a5f209a0fdd1e4" is_pre_election: true
I20260928 02:15:52.745810 26098 leader_election.cc:302] T f8cb44ccf21f41a3b15bfad27cac0fee P 748deb8124784323865761e33bb15a49 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 238723f3f6594ba69f78f47084af7896, 748deb8124784323865761e33bb15a49; no voters:
I20260928 02:15:52.746140 26033 raft_consensus.cc:2454] T f8cb44ccf21f41a3b15bfad27cac0fee P f641bde7b5f2435aa4a5f209a0fdd1e4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 748deb8124784323865761e33bb15a49 in term 0.
I20260928 02:15:52.746238 26363 raft_consensus.cc:2781] T f8cb44ccf21f41a3b15bfad27cac0fee P 748deb8124784323865761e33bb15a49 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 02:15:52.746409 26363 raft_consensus.cc:482] T f8cb44ccf21f41a3b15bfad27cac0fee P 748deb8124784323865761e33bb15a49 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 02:15:52.746502 26363 raft_consensus.cc:3037] T f8cb44ccf21f41a3b15bfad27cac0fee P 748deb8124784323865761e33bb15a49 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:15:52.749179 26363 raft_consensus.cc:504] T f8cb44ccf21f41a3b15bfad27cac0fee P 748deb8124784323865761e33bb15a49 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "238723f3f6594ba69f78f47084af7896" member_type: VOTER last_known_addr { host: "127.6.235.67" port: 34931 } } peers { permanent_uuid: "748deb8124784323865761e33bb15a49" member_type: VOTER last_known_addr { host: "127.6.235.66" port: 35753 } } peers { permanent_uuid: "f641bde7b5f2435aa4a5f209a0fdd1e4" member_type: VOTER last_known_addr { host: "127.6.235.65" port: 34977 } }
I20260928 02:15:52.750095 26363 leader_election.cc:288] T f8cb44ccf21f41a3b15bfad27cac0fee P 748deb8124784323865761e33bb15a49 [CANDIDATE]: Term 1 election: Requested vote from peers 238723f3f6594ba69f78f47084af7896 (127.6.235.67:34931), f641bde7b5f2435aa4a5f209a0fdd1e4 (127.6.235.65:34977)
I20260928 02:15:52.750635 26294 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f8cb44ccf21f41a3b15bfad27cac0fee" candidate_uuid: "748deb8124784323865761e33bb15a49" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "238723f3f6594ba69f78f47084af7896"
I20260928 02:15:52.750793 26033 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f8cb44ccf21f41a3b15bfad27cac0fee" candidate_uuid: "748deb8124784323865761e33bb15a49" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f641bde7b5f2435aa4a5f209a0fdd1e4"
I20260928 02:15:52.750895 26294 raft_consensus.cc:3037] T f8cb44ccf21f41a3b15bfad27cac0fee P 238723f3f6594ba69f78f47084af7896 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:15:52.751075 26033 raft_consensus.cc:3037] T f8cb44ccf21f41a3b15bfad27cac0fee P f641bde7b5f2435aa4a5f209a0fdd1e4 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:15:52.753368 26294 raft_consensus.cc:2454] T f8cb44ccf21f41a3b15bfad27cac0fee P 238723f3f6594ba69f78f47084af7896 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 748deb8124784323865761e33bb15a49 in term 1.
I20260928 02:15:52.753662 26033 raft_consensus.cc:2454] T f8cb44ccf21f41a3b15bfad27cac0fee P f641bde7b5f2435aa4a5f209a0fdd1e4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 748deb8124784323865761e33bb15a49 in term 1.
I20260928 02:15:52.754180 26098 leader_election.cc:302] T f8cb44ccf21f41a3b15bfad27cac0fee P 748deb8124784323865761e33bb15a49 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 748deb8124784323865761e33bb15a49, f641bde7b5f2435aa4a5f209a0fdd1e4; no voters:
I20260928 02:15:52.754535 26363 raft_consensus.cc:2781] T f8cb44ccf21f41a3b15bfad27cac0fee P 748deb8124784323865761e33bb15a49 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:15:52.755301 26363 raft_consensus.cc:686] T f8cb44ccf21f41a3b15bfad27cac0fee P 748deb8124784323865761e33bb15a49 [term 1 LEADER]: Becoming Leader. State: Replica: 748deb8124784323865761e33bb15a49, State: Running, Role: LEADER
I20260928 02:15:52.755951 26363 consensus_queue.cc:237] T f8cb44ccf21f41a3b15bfad27cac0fee P 748deb8124784323865761e33bb15a49 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "238723f3f6594ba69f78f47084af7896" member_type: VOTER last_known_addr { host: "127.6.235.67" port: 34931 } } peers { permanent_uuid: "748deb8124784323865761e33bb15a49" member_type: VOTER last_known_addr { host: "127.6.235.66" port: 35753 } } peers { permanent_uuid: "f641bde7b5f2435aa4a5f209a0fdd1e4" member_type: VOTER last_known_addr { host: "127.6.235.65" port: 34977 } }
I20260928 02:15:52.761351 25892 catalog_manager.cc:5495] T f8cb44ccf21f41a3b15bfad27cac0fee P 748deb8124784323865761e33bb15a49 reported cstate change: term changed from 0 to 1, leader changed from <none> to 748deb8124784323865761e33bb15a49 (127.6.235.66). New cstate: current_term: 1 leader_uuid: "748deb8124784323865761e33bb15a49" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "238723f3f6594ba69f78f47084af7896" member_type: VOTER last_known_addr { host: "127.6.235.67" port: 34931 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "748deb8124784323865761e33bb15a49" member_type: VOTER last_known_addr { host: "127.6.235.66" port: 35753 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f641bde7b5f2435aa4a5f209a0fdd1e4" member_type: VOTER last_known_addr { host: "127.6.235.65" port: 34977 } health_report { overall_health: UNKNOWN } } }
I20260928 02:15:52.781857 26354 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:15:52.775556 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37766 (local address 127.6.235.66:35753)
0928 02:15:52.775741 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0928 02:15:52.775745 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:15:52.775913 (+ 168us) server_negotiation.cc:407] Connection header received
0928 02:15:52.776151 (+ 238us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:15:52.776155 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:15:52.776253 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:15:52.776383 (+ 130us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 02:15:52.776770 (+ 387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:15:52.777648 (+ 878us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:15:52.779251 (+ 1603us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:15:52.779725 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:15:52.779821 (+ 96us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:15:52.780481 (+ 660us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 02:15:52.781262 (+ 781us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 02:15:52.781424 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0928 02:15:52.781612 (+ 188us) server_negotiation.cc:299] Negotiation successful
0928 02:15:52.781652 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260928 02:15:52.791388 26294 raft_consensus.cc:1261] T f8cb44ccf21f41a3b15bfad27cac0fee P 238723f3f6594ba69f78f47084af7896 [term 1 FOLLOWER]: Refusing update from remote peer 748deb8124784323865761e33bb15a49: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 02:15:52.791440 26033 raft_consensus.cc:1261] T f8cb44ccf21f41a3b15bfad27cac0fee P f641bde7b5f2435aa4a5f209a0fdd1e4 [term 1 FOLLOWER]: Refusing update from remote peer 748deb8124784323865761e33bb15a49: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 02:15:52.792505 26363 consensus_queue.cc:1035] T f8cb44ccf21f41a3b15bfad27cac0fee P 748deb8124784323865761e33bb15a49 [LEADER]: Connected to new peer: Peer: permanent_uuid: "238723f3f6594ba69f78f47084af7896" member_type: VOTER last_known_addr { host: "127.6.235.67" port: 34931 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 02:15:52.792879 26368 consensus_queue.cc:1035] T f8cb44ccf21f41a3b15bfad27cac0fee P 748deb8124784323865761e33bb15a49 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f641bde7b5f2435aa4a5f209a0fdd1e4" member_type: VOTER last_known_addr { host: "127.6.235.65" port: 34977 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 02:15:52.806106 26371 mvcc.cc:205] Tried to move back new op lower bound from 7334140939421900800 to 7334140939292872704. Current Snapshot: MvccSnapshot[applied={T|T < 7334140939421900800}]
I20260928 02:15:52.808032 26373 mvcc.cc:205] Tried to move back new op lower bound from 7334140939421900800 to 7334140939292872704. Current Snapshot: MvccSnapshot[applied={T|T < 7334140939421900800}]
I20260928 02:15:52.823175 26372 mvcc.cc:205] Tried to move back new op lower bound from 7334140939421900800 to 7334140939292872704. Current Snapshot: MvccSnapshot[applied={T|T < 7334140939421900800}]
I20260928 02:15:52.834825 25892 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:41828:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 02:15:52.835139 25892 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:41828:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 02:15:52.839107 25892 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 9d95ab259c9b4bd6899fd454942615b5: Sending DeleteTablet for 3 replicas of tablet f8cb44ccf21f41a3b15bfad27cac0fee
I20260928 02:15:52.840412 26274 tablet_service.cc:1514] Processing DeleteTablet for tablet f8cb44ccf21f41a3b15bfad27cac0fee with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:15:52 UTC) from {username='slave'} at 127.0.0.1:48032
I20260928 02:15:52.840678 26144 tablet_service.cc:1514] Processing DeleteTablet for tablet f8cb44ccf21f41a3b15bfad27cac0fee with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:15:52 UTC) from {username='slave'} at 127.0.0.1:37764
I20260928 02:15:52.841663 7085 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskTlaUuc/build/asan/bin/kudu pid 25951
I20260928 02:15:52.841560 26013 tablet_service.cc:1514] Processing DeleteTablet for tablet f8cb44ccf21f41a3b15bfad27cac0fee with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:15:52 UTC) from {username='slave'} at 127.0.0.1:54374
I20260928 02:15:52.842796 26381 tablet_replica.cc:330] T f8cb44ccf21f41a3b15bfad27cac0fee P 238723f3f6594ba69f78f47084af7896: stopping tablet replica
I20260928 02:15:52.843271 26381 raft_consensus.cc:2229] T f8cb44ccf21f41a3b15bfad27cac0fee P 238723f3f6594ba69f78f47084af7896 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 02:15:52.843771 26381 raft_consensus.cc:2258] T f8cb44ccf21f41a3b15bfad27cac0fee P 238723f3f6594ba69f78f47084af7896 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 02:15:52.846504 26381 ts_tablet_manager.cc:1905] T f8cb44ccf21f41a3b15bfad27cac0fee P 238723f3f6594ba69f78f47084af7896: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 02:15:52.844841 26383 tablet_replica.cc:330] T f8cb44ccf21f41a3b15bfad27cac0fee P f641bde7b5f2435aa4a5f209a0fdd1e4: stopping tablet replica
I20260928 02:15:52.849413 26383 raft_consensus.cc:2229] T f8cb44ccf21f41a3b15bfad27cac0fee P f641bde7b5f2435aa4a5f209a0fdd1e4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 02:15:52.849967 26383 raft_consensus.cc:2258] T f8cb44ccf21f41a3b15bfad27cac0fee P f641bde7b5f2435aa4a5f209a0fdd1e4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 02:15:52.855873 26382 tablet_replica.cc:330] T f8cb44ccf21f41a3b15bfad27cac0fee P 748deb8124784323865761e33bb15a49: stopping tablet replica
I20260928 02:15:52.856516 26382 raft_consensus.cc:2229] T f8cb44ccf21f41a3b15bfad27cac0fee P 748deb8124784323865761e33bb15a49 [term 1 LEADER]: Raft consensus shutting down.
I20260928 02:15:52.857290 26355 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:15:52.844822 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54390 (local address 127.6.235.65:34977)
0928 02:15:52.845017 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0928 02:15:52.845023 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 02:15:52.846548 (+ 1525us) server_negotiation.cc:407] Connection header received
0928 02:15:52.846943 (+ 395us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:15:52.846951 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:15:52.847111 (+ 160us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:15:52.847230 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:15:52.848115 (+ 885us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:15:52.849686 (+ 1571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:15:52.852282 (+ 2596us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:15:52.853239 (+ 957us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:15:52.853368 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:15:52.855306 (+ 1938us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:15:52.855317 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:15:52.855335 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 02:15:52.856254 (+ 919us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:15:52.856317 (+ 63us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 02:15:52.856342 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:15:52.856437 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0928 02:15:52.856740 (+ 303us) server_negotiation.cc:299] Negotiation successful
0928 02:15:52.856856 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260928 02:15:52.856880 26383 ts_tablet_manager.cc:1905] T f8cb44ccf21f41a3b15bfad27cac0fee P f641bde7b5f2435aa4a5f209a0fdd1e4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 02:15:52.857335 26382 raft_consensus.cc:2258] T f8cb44ccf21f41a3b15bfad27cac0fee P 748deb8124784323865761e33bb15a49 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 02:15:52.858129 26073 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 02:15:52.858939 26381 ts_tablet_manager.cc:1918] T f8cb44ccf21f41a3b15bfad27cac0fee P 238723f3f6594ba69f78f47084af7896: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 02:15:52.859127 26381 log.cc:1196] T f8cb44ccf21f41a3b15bfad27cac0fee P 238723f3f6594ba69f78f47084af7896: Deleting WAL directory at /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790561432251112-7085-0/minicluster-data/ts-2/wal/wals/f8cb44ccf21f41a3b15bfad27cac0fee
I20260928 02:15:52.859711 26381 ts_tablet_manager.cc:1939] T f8cb44ccf21f41a3b15bfad27cac0fee P 238723f3f6594ba69f78f47084af7896: Deleting consensus metadata
I20260928 02:15:52.860148 26382 ts_tablet_manager.cc:1905] T f8cb44ccf21f41a3b15bfad27cac0fee P 748deb8124784323865761e33bb15a49: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 02:15:52.861699 25877 catalog_manager.cc:4841] TS 238723f3f6594ba69f78f47084af7896 (127.6.235.67:34931): tablet f8cb44ccf21f41a3b15bfad27cac0fee (table test-table [id=4550152b62134a72913b2fb65ef23d98]) successfully deleted
I20260928 02:15:52.869280 26382 ts_tablet_manager.cc:1918] T f8cb44ccf21f41a3b15bfad27cac0fee P 748deb8124784323865761e33bb15a49: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 02:15:52.869501 26382 log.cc:1196] T f8cb44ccf21f41a3b15bfad27cac0fee P 748deb8124784323865761e33bb15a49: Deleting WAL directory at /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790561432251112-7085-0/minicluster-data/ts-1/wal/wals/f8cb44ccf21f41a3b15bfad27cac0fee
I20260928 02:15:52.870210 26382 ts_tablet_manager.cc:1939] T f8cb44ccf21f41a3b15bfad27cac0fee P 748deb8124784323865761e33bb15a49: Deleting consensus metadata
I20260928 02:15:52.872447 25877 catalog_manager.cc:4841] TS 748deb8124784323865761e33bb15a49 (127.6.235.66:35753): tablet f8cb44ccf21f41a3b15bfad27cac0fee (table test-table [id=4550152b62134a72913b2fb65ef23d98]) successfully deleted
I20260928 02:15:52.957072 7085 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTlaUuc/build/asan/bin/kudu with pid 25951
W20260928 02:15:52.983189 25877 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.6.235.65:34977 (error 108)
I20260928 02:15:52.983956 7085 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskTlaUuc/build/asan/bin/kudu pid 26082
I20260928 02:15:52.986392 26356 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:15:52.985793 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.235.65:34977 (local address 127.0.0.1:54402)
0928 02:15:52.986072 (+ 279us) negotiation.cc:107] Waiting for socket to connect
0928 02:15:52.986210 (+ 138us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.235.65:34977: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":139}
W20260928 02:15:52.986682 25877 catalog_manager.cc:4584] TS f641bde7b5f2435aa4a5f209a0fdd1e4 (127.6.235.65:34977): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet f8cb44ccf21f41a3b15bfad27cac0fee: Network error: Client connection negotiation failed: client connection to 127.6.235.65:34977: connect: Connection refused (error 111)
I20260928 02:15:52.992933 26354 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:15:52.985267 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37770 (local address 127.6.235.66:35753)
0928 02:15:52.985444 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0928 02:15:52.985450 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 02:15:52.985471 (+ 21us) server_negotiation.cc:407] Connection header received
0928 02:15:52.985536 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:15:52.985542 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:15:52.985664 (+ 122us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:15:52.985777 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:15:52.986311 (+ 534us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:15:52.987513 (+ 1202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:15:52.988863 (+ 1350us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:15:52.989578 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:15:52.989735 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:15:52.991128 (+ 1393us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:15:52.991141 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:15:52.991161 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 02:15:52.992010 (+ 849us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:15:52.992075 (+ 65us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 02:15:52.992100 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:15:52.992232 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0928 02:15:52.992401 (+ 169us) server_negotiation.cc:299] Negotiation successful
0928 02:15:52.992505 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260928 02:15:52.993750 26204 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 02:15:53.040450 26356 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:15:53.039783 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.235.65:34977 (local address 127.0.0.1:54404)
0928 02:15:53.040141 (+ 358us) negotiation.cc:107] Waiting for socket to connect
0928 02:15:53.040257 (+ 116us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.235.65:34977: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":97}
I20260928 02:15:53.041710 26356 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:15:53.041281 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.235.65:34977 (local address 127.0.0.1:54418)
0928 02:15:53.041455 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0928 02:15:53.041550 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.235.65:34977: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
I20260928 02:15:53.102445 26356 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:15:53.101864 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.235.65:34977 (local address 127.0.0.1:54424)
0928 02:15:53.102127 (+ 263us) negotiation.cc:107] Waiting for socket to connect
0928 02:15:53.102230 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.235.65:34977: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":103}
I20260928 02:15:53.103564 26356 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:15:53.103168 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.235.65:34977 (local address 127.0.0.1:54432)
0928 02:15:53.103336 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0928 02:15:53.103427 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.235.65:34977: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
I20260928 02:15:53.111253 7085 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTlaUuc/build/asan/bin/kudu with pid 26082
I20260928 02:15:53.131750 7085 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskTlaUuc/build/asan/bin/kudu pid 26212
I20260928 02:15:53.139730 26352 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:15:53.132630 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48042 (local address 127.6.235.67:34931)
0928 02:15:53.132860 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0928 02:15:53.132868 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 02:15:53.132918 (+ 50us) server_negotiation.cc:407] Connection header received
0928 02:15:53.133384 (+ 466us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:15:53.133403 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:15:53.133508 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:15:53.133635 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:15:53.134099 (+ 464us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:15:53.134932 (+ 833us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:15:53.136321 (+ 1389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:15:53.136751 (+ 430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:15:53.136958 (+ 207us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:15:53.138287 (+ 1329us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:15:53.138298 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:15:53.138315 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 02:15:53.138910 (+ 595us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:15:53.138966 (+ 56us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 02:15:53.138988 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:15:53.139129 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0928 02:15:53.139254 (+ 125us) server_negotiation.cc:299] Negotiation successful
0928 02:15:53.139372 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
I20260928 02:15:53.140461 26334 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 02:15:53.184144 26356 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:15:53.183549 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.235.65:34977 (local address 127.0.0.1:54436)
0928 02:15:53.183830 (+ 281us) negotiation.cc:107] Waiting for socket to connect
0928 02:15:53.183947 (+ 117us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.235.65:34977: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":120}
I20260928 02:15:53.185854 26356 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:15:53.185376 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.235.65:34977 (local address 127.0.0.1:54446)
0928 02:15:53.185628 (+ 252us) negotiation.cc:107] Waiting for socket to connect
0928 02:15:53.185715 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.235.65:34977: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":135}
I20260928 02:15:53.233827 7085 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTlaUuc/build/asan/bin/kudu with pid 26212
I20260928 02:15:53.253551 7085 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskTlaUuc/build/asan/bin/kudu pid 25861
I20260928 02:15:53.255046 25922 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 02:15:53.367841 26356 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:15:53.367164 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.235.65:34977 (local address 127.0.0.1:54454)
0928 02:15:53.367361 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0928 02:15:53.367476 (+ 115us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.235.65:34977: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
I20260928 02:15:53.368387 7085 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTlaUuc/build/asan/bin/kudu with pid 25861
2026-09-28T02:15:53Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/3 (3117 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/4
Loading random data
Initializing database '/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790561432251112-7085-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 02:15:53 dist-test-slave-86gv krb5kdc[26392](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 02:15:53 dist-test-slave-86gv krb5kdc[26392](info): set up 2 sockets
Sep 28 02:15:53 dist-test-slave-86gv krb5kdc[26392](info): commencing operation
krb5kdc: starting...
W20260928 02:15:54.501904 7085 mini_kdc.cc:121] Time spent starting KDC: real 1.090s user 0.001s sys 0.028s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790561432251112-7085-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790561432251112-7085-0/krb5kdc/test-user.keytab.
Sep 28 02:15:54 dist-test-slave-86gv krb5kdc[26392](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561754, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T02:15:54Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T02:15:54Z Disabled control of system clock
WARNING: no policy specified for kudu/127.6.235.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.235.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.235.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790561432251112-7085-0/krb5kdc/kudu_127.6.235.126.keytab.
Entry for principal kudu/127.6.235.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790561432251112-7085-0/krb5kdc/kudu_127.6.235.126.keytab.
I20260928 02:15:54.785652 7085 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTlaUuc/build/asan/bin/kudu
/tmp/dist-test-taskTlaUuc/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790561432251112-7085-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790561432251112-7085-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790561432251112-7085-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790561432251112-7085-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.235.126:36891
--webserver_interface=127.6.235.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790561432251112-7085-0/krb5kdc/kudu_127.6.235.126.keytab
--principal=kudu/127.6.235.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.235.84:36435
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.235.126:36891
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790561432251112-7085-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790561432251112-7085-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790561432251112-7085-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:15:55.140614 26405 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:15:55.141032 26405 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:15:55.141218 26405 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:15:55.149027 26405 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 02:15:55.149151 26405 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:15:55.149223 26405 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:15:55.149317 26405 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 02:15:55.149396 26405 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 02:15:55.160637 26405 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.235.84:36435
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790561432251112-7085-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790561432251112-7085-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.235.126:36891
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790561432251112-7085-0/krb5kdc/kudu_127.6.235.126.keytab
--principal=kudu/127.6.235.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.235.126:36891
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790561432251112-7085-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.235.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790561432251112-7085-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 00:51:59 UTC on ccycloud.quasar-axgkwn.root.comops.site
build id 4676464
ASAN enabled
I20260928 02:15:55.162240 26405 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:15:55.163725 26405 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:15:55.176656 26410 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:15:55.176558 26413 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:15:55.176633 26411 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:15:55.177645 26405 server_base.cc:1034] running on GCE node
I20260928 02:15:55.178580 26405 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:15:55.180191 26405 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:15:55.181447 26405 hybrid_clock.cc:648] HybridClock initialized: now 1790561755181409 us; error 45 us; skew 500 ppm
I20260928 02:15:55.181880 26405 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 02:15:55 dist-test-slave-86gv krb5kdc[26392](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561755, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.235.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:15:55.192873 26405 init.cc:377] Logged in from keytab as kudu/127.6.235.126@KRBTEST.COM (short username kudu)
I20260928 02:15:55.194906 26405 webserver.cc:460] Webserver started at http://127.6.235.126:37821/ using document root <none> and password file <none>
I20260928 02:15:55.195724 26405 fs_manager.cc:359] Metadata directory not provided
I20260928 02:15:55.195891 26405 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:15:55.196230 26405 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:15:55.199694 26405 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790561432251112-7085-0/minicluster-data/master-0/data/instance:
uuid: "68fabc3a8fd74b1ea6a30861cf2178e9"
format_stamp: "Formatted at 2026-09-28 02:15:55 on dist-test-slave-86gv"
server_key: "ff6ec8d6342333c63c9cdfc3c2e12aa3"
server_key_iv: "b4fa7b429503ee78c9f753d79921701b"
server_key_version: "encryptionkey@0"
I20260928 02:15:55.200562 26405 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790561432251112-7085-0/minicluster-data/master-0/wal/instance:
uuid: "68fabc3a8fd74b1ea6a30861cf2178e9"
format_stamp: "Formatted at 2026-09-28 02:15:55 on dist-test-slave-86gv"
server_key: "ff6ec8d6342333c63c9cdfc3c2e12aa3"
server_key_iv: "b4fa7b429503ee78c9f753d79921701b"
server_key_version: "encryptionkey@0"
I20260928 02:15:55.206409 26405 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260928 02:15:55.210390 26420 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:15:55.212180 26405 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260928 02:15:55.212395 26405 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790561432251112-7085-0/minicluster-data/master-0/data,/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790561432251112-7085-0/minicluster-data/master-0/wal
uuid: "68fabc3a8fd74b1ea6a30861cf2178e9"
format_stamp: "Formatted at 2026-09-28 02:15:55 on dist-test-slave-86gv"
server_key: "ff6ec8d6342333c63c9cdfc3c2e12aa3"
server_key_iv: "b4fa7b429503ee78c9f753d79921701b"
server_key_version: "encryptionkey@0"
I20260928 02:15:55.212628 26405 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790561432251112-7085-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790561432251112-7085-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790561432251112-7085-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:15:55.234555 26405 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:15:55.239377 26405 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:15:55.239691 26405 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:15:55.259266 26405 rpc_server.cc:307] RPC server started. Bound to: 127.6.235.126:36891
I20260928 02:15:55.259287 26471 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.235.126:36891 every 8 connection(s)
I20260928 02:15:55.260731 26405 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790561432251112-7085-0/minicluster-data/master-0/data/info.pb
I20260928 02:15:55.263208 7085 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTlaUuc/build/asan/bin/kudu as pid 26405
I20260928 02:15:55.263410 7085 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790561432251112-7085-0/minicluster-data/master-0/wal/instance
I20260928 02:15:55.263949 7085 external_mini_cluster.cc:1428] Setting key d544e2fc1e0919ec16b6f5e9e8cb0089
I20260928 02:15:55.265574 26472 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 28 02:15:55 dist-test-slave-86gv krb5kdc[26392](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561754, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.235.126@KRBTEST.COM
I20260928 02:15:55.280484 26472 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 68fabc3a8fd74b1ea6a30861cf2178e9: Bootstrap starting.
I20260928 02:15:55.285080 26472 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 68fabc3a8fd74b1ea6a30861cf2178e9: Neither blocks nor log segments found. Creating new log.
I20260928 02:15:55.286406 26472 log.cc:824] T 00000000000000000000000000000000 P 68fabc3a8fd74b1ea6a30861cf2178e9: Log is configured to *not* fsync() on all Append() calls
I20260928 02:15:55.289918 26475 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:15:55.265974 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47676 (local address 127.6.235.126:36891)
0928 02:15:55.266789 (+ 815us) server_negotiation.cc:206] Beginning negotiation
0928 02:15:55.266802 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0928 02:15:55.266846 (+ 44us) server_negotiation.cc:407] Connection header received
0928 02:15:55.268470 (+ 1624us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:15:55.268492 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:15:55.268943 (+ 451us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:15:55.269432 (+ 489us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:15:55.271251 (+ 1819us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:15:55.272697 (+ 1446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:15:55.274564 (+ 1867us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:15:55.275470 (+ 906us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:15:55.275637 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:15:55.280709 (+ 5072us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:15:55.280720 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:15:55.280741 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:15:55.280795 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:15:55.284809 (+ 4014us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:15:55.285757 (+ 948us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:15:55.285767 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:15:55.285777 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:15:55.285990 (+ 213us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:15:55.286425 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:15:55.286431 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:15:55.286434 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:15:55.287202 (+ 768us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:15:55.287461 (+ 259us) server_negotiation.cc:1035] Waiting for connection context
0928 02:15:55.287985 (+ 524us) server_negotiation.cc:299] Negotiation successful
0928 02:15:55.288499 (+ 514us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":614,"thread_start_us":352,"threads_started":1}
I20260928 02:15:55.290205 26472 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 68fabc3a8fd74b1ea6a30861cf2178e9: No bootstrap required, opened a new log
I20260928 02:15:55.298365 26472 raft_consensus.cc:348] T 00000000000000000000000000000000 P 68fabc3a8fd74b1ea6a30861cf2178e9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "68fabc3a8fd74b1ea6a30861cf2178e9" member_type: VOTER last_known_addr { host: "127.6.235.126" port: 36891 } }
I20260928 02:15:55.298852 26472 raft_consensus.cc:374] T 00000000000000000000000000000000 P 68fabc3a8fd74b1ea6a30861cf2178e9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:15:55.298957 26472 raft_consensus.cc:729] T 00000000000000000000000000000000 P 68fabc3a8fd74b1ea6a30861cf2178e9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 68fabc3a8fd74b1ea6a30861cf2178e9, State: Initialized, Role: FOLLOWER
I20260928 02:15:55.299671 26472 consensus_queue.cc:260] T 00000000000000000000000000000000 P 68fabc3a8fd74b1ea6a30861cf2178e9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "68fabc3a8fd74b1ea6a30861cf2178e9" member_type: VOTER last_known_addr { host: "127.6.235.126" port: 36891 } }
I20260928 02:15:55.299942 26472 raft_consensus.cc:388] T 00000000000000000000000000000000 P 68fabc3a8fd74b1ea6a30861cf2178e9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 02:15:55.300091 26472 raft_consensus.cc:482] T 00000000000000000000000000000000 P 68fabc3a8fd74b1ea6a30861cf2178e9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 02:15:55.300271 26472 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 68fabc3a8fd74b1ea6a30861cf2178e9 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:15:55.302628 26472 raft_consensus.cc:504] T 00000000000000000000000000000000 P 68fabc3a8fd74b1ea6a30861cf2178e9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "68fabc3a8fd74b1ea6a30861cf2178e9" member_type: VOTER last_known_addr { host: "127.6.235.126" port: 36891 } }
I20260928 02:15:55.303180 26472 leader_election.cc:302] T 00000000000000000000000000000000 P 68fabc3a8fd74b1ea6a30861cf2178e9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 68fabc3a8fd74b1ea6a30861cf2178e9; no voters:
I20260928 02:15:55.303807 26472 leader_election.cc:288] T 00000000000000000000000000000000 P 68fabc3a8fd74b1ea6a30861cf2178e9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 02:15:55.303962 26477 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 68fabc3a8fd74b1ea6a30861cf2178e9 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:15:55.304622 26477 raft_consensus.cc:686] T 00000000000000000000000000000000 P 68fabc3a8fd74b1ea6a30861cf2178e9 [term 1 LEADER]: Becoming Leader. State: Replica: 68fabc3a8fd74b1ea6a30861cf2178e9, State: Running, Role: LEADER
I20260928 02:15:55.305267 26477 consensus_queue.cc:237] T 00000000000000000000000000000000 P 68fabc3a8fd74b1ea6a30861cf2178e9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "68fabc3a8fd74b1ea6a30861cf2178e9" member_type: VOTER last_known_addr { host: "127.6.235.126" port: 36891 } }
I20260928 02:15:55.305979 26472 sys_catalog.cc:564] T 00000000000000000000000000000000 P 68fabc3a8fd74b1ea6a30861cf2178e9 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 02:15:55.308616 26479 sys_catalog.cc:455] T 00000000000000000000000000000000 P 68fabc3a8fd74b1ea6a30861cf2178e9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 68fabc3a8fd74b1ea6a30861cf2178e9. Latest consensus state: current_term: 1 leader_uuid: "68fabc3a8fd74b1ea6a30861cf2178e9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "68fabc3a8fd74b1ea6a30861cf2178e9" member_type: VOTER last_known_addr { host: "127.6.235.126" port: 36891 } } }
I20260928 02:15:55.308642 26478 sys_catalog.cc:455] T 00000000000000000000000000000000 P 68fabc3a8fd74b1ea6a30861cf2178e9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "68fabc3a8fd74b1ea6a30861cf2178e9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "68fabc3a8fd74b1ea6a30861cf2178e9" member_type: VOTER last_known_addr { host: "127.6.235.126" port: 36891 } } }
I20260928 02:15:55.309177 26478 sys_catalog.cc:458] T 00000000000000000000000000000000 P 68fabc3a8fd74b1ea6a30861cf2178e9 [sys.catalog]: This master's current role is: LEADER
I20260928 02:15:55.309177 26479 sys_catalog.cc:458] T 00000000000000000000000000000000 P 68fabc3a8fd74b1ea6a30861cf2178e9 [sys.catalog]: This master's current role is: LEADER
I20260928 02:15:55.310719 26484 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 02:15:55.316937 26484 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 02:15:55.326910 26484 catalog_manager.cc:1347] Generated new cluster ID: 3d9771743e3e4af7a864b02f405f8e51
I20260928 02:15:55.327045 26484 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 02:15:55.360093 26484 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 02:15:55.362195 26484 catalog_manager.cc:1504] Loading token signing keys...
I20260928 02:15:55.378022 26484 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 68fabc3a8fd74b1ea6a30861cf2178e9: Generated new TSK 0
I20260928 02:15:55.379076 26484 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.6.235.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.235.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.235.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790561432251112-7085-0/krb5kdc/kudu_127.6.235.65.keytab.
Entry for principal kudu/127.6.235.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790561432251112-7085-0/krb5kdc/kudu_127.6.235.65.keytab.
I20260928 02:15:55.457676 7085 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTlaUuc/build/asan/bin/kudu
/tmp/dist-test-taskTlaUuc/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790561432251112-7085-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790561432251112-7085-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790561432251112-7085-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790561432251112-7085-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.235.65:0
--local_ip_for_outbound_sockets=127.6.235.65
--webserver_interface=127.6.235.65
--webserver_port=0
--tserver_master_addrs=127.6.235.126:36891
--keytab_file=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790561432251112-7085-0/krb5kdc/kudu_127.6.235.65.keytab
--principal=kudu/127.6.235.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.235.84:36435
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790561432251112-7085-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790561432251112-7085-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790561432251112-7085-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:15:55.822465 26498 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:15:55.822779 26498 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:15:55.823021 26498 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:15:55.831115 26498 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:15:55.831257 26498 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:15:55.831377 26498 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.235.65
I20260928 02:15:55.842304 26498 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.235.84:36435
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790561432251112-7085-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790561432251112-7085-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790561432251112-7085-0/krb5kdc/kudu_127.6.235.65.keytab
--principal=kudu/127.6.235.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.235.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790561432251112-7085-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.6.235.65
--webserver_port=0
--tserver_master_addrs=127.6.235.126:36891
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.235.65
--log_dir=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790561432251112-7085-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 00:51:59 UTC on ccycloud.quasar-axgkwn.root.comops.site
build id 4676464
ASAN enabled
I20260928 02:15:55.843743 26498 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:15:55.845122 26498 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:15:55.859171 26504 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:15:55.859244 26503 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:15:55.859205 26506 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:15:55.860059 26498 server_base.cc:1034] running on GCE node
I20260928 02:15:55.860788 26498 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:15:55.862059 26498 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:15:55.863292 26498 hybrid_clock.cc:648] HybridClock initialized: now 1790561755863138 us; error 148 us; skew 500 ppm
I20260928 02:15:55.863730 26498 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 02:15:55 dist-test-slave-86gv krb5kdc[26392](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561755, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.235.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:15:55.867311 26498 init.cc:377] Logged in from keytab as kudu/127.6.235.65@KRBTEST.COM (short username kudu)
I20260928 02:15:55.869092 26498 webserver.cc:460] Webserver started at http://127.6.235.65:39643/ using document root <none> and password file <none>
I20260928 02:15:55.869844 26498 fs_manager.cc:359] Metadata directory not provided
I20260928 02:15:55.869978 26498 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:15:55.870321 26498 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:15:55.873539 26498 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790561432251112-7085-0/minicluster-data/ts-0/data/instance:
uuid: "05bc56491dbc468fbab24c4367a76967"
format_stamp: "Formatted at 2026-09-28 02:15:55 on dist-test-slave-86gv"
server_key: "b8b86d4d19e076d4b9ef83f890b503fa"
server_key_iv: "d6c2a3b8c491dffed214427c5389eb03"
server_key_version: "encryptionkey@0"
I20260928 02:15:55.874527 26498 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790561432251112-7085-0/minicluster-data/ts-0/wal/instance:
uuid: "05bc56491dbc468fbab24c4367a76967"
format_stamp: "Formatted at 2026-09-28 02:15:55 on dist-test-slave-86gv"
server_key: "b8b86d4d19e076d4b9ef83f890b503fa"
server_key_iv: "d6c2a3b8c491dffed214427c5389eb03"
server_key_version: "encryptionkey@0"
I20260928 02:15:55.880553 26498 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.001s
I20260928 02:15:55.884652 26513 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:15:55.887001 26498 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260928 02:15:55.887277 26498 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790561432251112-7085-0/minicluster-data/ts-0/data,/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790561432251112-7085-0/minicluster-data/ts-0/wal
uuid: "05bc56491dbc468fbab24c4367a76967"
format_stamp: "Formatted at 2026-09-28 02:15:55 on dist-test-slave-86gv"
server_key: "b8b86d4d19e076d4b9ef83f890b503fa"
server_key_iv: "d6c2a3b8c491dffed214427c5389eb03"
server_key_version: "encryptionkey@0"
I20260928 02:15:55.887540 26498 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790561432251112-7085-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790561432251112-7085-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790561432251112-7085-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:15:55.904552 26498 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:15:55.909206 26498 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:15:55.909511 26498 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:15:55.910713 26498 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:15:55.912608 26498 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:15:55.912684 26498 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:15:55.912811 26498 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:15:55.912878 26498 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:15:55.948768 26498 rpc_server.cc:307] RPC server started. Bound to: 127.6.235.65:43845
I20260928 02:15:55.948819 26625 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.235.65:43845 every 8 connection(s)
I20260928 02:15:55.950203 26498 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790561432251112-7085-0/minicluster-data/ts-0/data/info.pb
I20260928 02:15:55.955837 7085 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTlaUuc/build/asan/bin/kudu as pid 26498
I20260928 02:15:55.956027 7085 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790561432251112-7085-0/minicluster-data/ts-0/wal/instance
I20260928 02:15:55.956493 7085 external_mini_cluster.cc:1428] Setting key 9292476733ca5cfe93c5a9d2ba9f29d0
Sep 28 02:15:55 dist-test-slave-86gv krb5kdc[26392](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561755, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.235.65@KRBTEST.COM for kudu/127.6.235.126@KRBTEST.COM
I20260928 02:15:55.973484 26628 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:15:55.953159 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.235.65:49799 (local address 127.6.235.126:36891)
0928 02:15:55.953632 (+ 473us) server_negotiation.cc:206] Beginning negotiation
0928 02:15:55.953639 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 02:15:55.954791 (+ 1152us) server_negotiation.cc:407] Connection header received
0928 02:15:55.956935 (+ 2144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:15:55.956941 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:15:55.957046 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:15:55.957179 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:15:55.958961 (+ 1782us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:15:55.960192 (+ 1231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:15:55.961926 (+ 1734us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:15:55.962638 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:15:55.962784 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:15:55.966557 (+ 3773us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:15:55.966566 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:15:55.966575 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:15:55.966625 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:15:55.969636 (+ 3011us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:15:55.970765 (+ 1129us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:15:55.970770 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:15:55.970772 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:15:55.970903 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:15:55.971500 (+ 597us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:15:55.971505 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:15:55.971507 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:15:55.971796 (+ 289us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:15:55.971967 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0928 02:15:55.972971 (+ 1004us) server_negotiation.cc:299] Negotiation successful
0928 02:15:55.973226 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":346,"thread_start_us":223,"threads_started":1}
I20260928 02:15:55.975111 26629 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:15:55.953817 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.235.126:36891 (local address 127.6.235.65:49799)
0928 02:15:55.954592 (+ 775us) negotiation.cc:107] Waiting for socket to connect
0928 02:15:55.954643 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0928 02:15:55.956525 (+ 1882us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:15:55.957470 (+ 945us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:15:55.957479 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:15:55.958055 (+ 576us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:15:55.958783 (+ 728us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:15:55.958802 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:15:55.960364 (+ 1562us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:15:55.960368 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:15:55.961775 (+ 1407us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:15:55.961788 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:15:55.962871 (+ 1083us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:15:55.962876 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:15:55.963122 (+ 246us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:15:55.963899 (+ 777us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:15:55.963923 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:15:55.966274 (+ 2351us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:15:55.969808 (+ 3534us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:15:55.969821 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:15:55.969844 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:15:55.970562 (+ 718us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:15:55.971037 (+ 475us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:15:55.971045 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:15:55.971052 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:15:55.971340 (+ 288us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:15:55.971942 (+ 602us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:15:55.971954 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:15:55.972462 (+ 508us) client_negotiation.cc:769] Sending connection context
0928 02:15:55.972793 (+ 331us) client_negotiation.cc:240] Negotiation successful
0928 02:15:55.973472 (+ 679us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":491,"thread_start_us":287,"threads_started":1}
I20260928 02:15:55.977077 26626 heartbeater.cc:347] Connected to a master server at 127.6.235.126:36891
I20260928 02:15:55.977423 26626 heartbeater.cc:464] Registering TS with master...
I20260928 02:15:55.978302 26626 heartbeater.cc:511] Master 127.6.235.126:36891 requested a full tablet report, sending...
I20260928 02:15:55.980901 26437 ts_manager.cc:194] Registered new tserver with Master: 05bc56491dbc468fbab24c4367a76967 (127.6.235.65:43845)
I20260928 02:15:55.982558 26437 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.235.65@KRBTEST.COM'} at 127.6.235.65:49799
WARNING: no policy specified for kudu/127.6.235.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.235.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.235.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790561432251112-7085-0/krb5kdc/kudu_127.6.235.66.keytab.
Entry for principal kudu/127.6.235.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790561432251112-7085-0/krb5kdc/kudu_127.6.235.66.keytab.
I20260928 02:15:56.024955 7085 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTlaUuc/build/asan/bin/kudu
/tmp/dist-test-taskTlaUuc/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790561432251112-7085-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790561432251112-7085-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790561432251112-7085-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790561432251112-7085-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.235.66:0
--local_ip_for_outbound_sockets=127.6.235.66
--webserver_interface=127.6.235.66
--webserver_port=0
--tserver_master_addrs=127.6.235.126:36891
--keytab_file=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790561432251112-7085-0/krb5kdc/kudu_127.6.235.66.keytab
--principal=kudu/127.6.235.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.235.84:36435
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790561432251112-7085-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790561432251112-7085-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790561432251112-7085-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:15:56.386248 26632 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:15:56.386620 26632 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:15:56.386801 26632 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:15:56.394742 26632 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:15:56.394948 26632 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:15:56.395138 26632 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.235.66
I20260928 02:15:56.405805 26632 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.235.84:36435
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790561432251112-7085-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790561432251112-7085-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790561432251112-7085-0/krb5kdc/kudu_127.6.235.66.keytab
--principal=kudu/127.6.235.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.235.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790561432251112-7085-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.6.235.66
--webserver_port=0
--tserver_master_addrs=127.6.235.126:36891
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.235.66
--log_dir=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790561432251112-7085-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 00:51:59 UTC on ccycloud.quasar-axgkwn.root.comops.site
build id 4676464
ASAN enabled
I20260928 02:15:56.407441 26632 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:15:56.409000 26632 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:15:56.423888 26637 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:15:56.423913 26640 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:15:56.423862 26638 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:15:56.424935 26632 server_base.cc:1034] running on GCE node
I20260928 02:15:56.425559 26632 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:15:56.426513 26632 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:15:56.427727 26632 hybrid_clock.cc:648] HybridClock initialized: now 1790561756427665 us; error 67 us; skew 500 ppm
I20260928 02:15:56.428146 26632 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 02:15:56 dist-test-slave-86gv krb5kdc[26392](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561756, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.235.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:15:56.431617 26632 init.cc:377] Logged in from keytab as kudu/127.6.235.66@KRBTEST.COM (short username kudu)
I20260928 02:15:56.433226 26632 webserver.cc:460] Webserver started at http://127.6.235.66:38213/ using document root <none> and password file <none>
I20260928 02:15:56.433954 26632 fs_manager.cc:359] Metadata directory not provided
I20260928 02:15:56.434083 26632 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:15:56.434427 26632 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:15:56.437616 26632 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790561432251112-7085-0/minicluster-data/ts-1/data/instance:
uuid: "47cf21fa9bd045de8d3f13e19e60751a"
format_stamp: "Formatted at 2026-09-28 02:15:56 on dist-test-slave-86gv"
server_key: "d13e2503ca6f256c8a36375426314bd2"
server_key_iv: "cf61292c42f935446f825cee18699c5c"
server_key_version: "encryptionkey@0"
I20260928 02:15:56.438439 26632 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790561432251112-7085-0/minicluster-data/ts-1/wal/instance:
uuid: "47cf21fa9bd045de8d3f13e19e60751a"
format_stamp: "Formatted at 2026-09-28 02:15:56 on dist-test-slave-86gv"
server_key: "d13e2503ca6f256c8a36375426314bd2"
server_key_iv: "cf61292c42f935446f825cee18699c5c"
server_key_version: "encryptionkey@0"
I20260928 02:15:56.443915 26632 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260928 02:15:56.447680 26647 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:15:56.449357 26632 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 02:15:56.449560 26632 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790561432251112-7085-0/minicluster-data/ts-1/data,/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790561432251112-7085-0/minicluster-data/ts-1/wal
uuid: "47cf21fa9bd045de8d3f13e19e60751a"
format_stamp: "Formatted at 2026-09-28 02:15:56 on dist-test-slave-86gv"
server_key: "d13e2503ca6f256c8a36375426314bd2"
server_key_iv: "cf61292c42f935446f825cee18699c5c"
server_key_version: "encryptionkey@0"
I20260928 02:15:56.449784 26632 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790561432251112-7085-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790561432251112-7085-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790561432251112-7085-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:15:56.485742 26632 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:15:56.490069 26632 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:15:56.490375 26632 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:15:56.491425 26632 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:15:56.493027 26632 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:15:56.493125 26632 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:15:56.493263 26632 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:15:56.493332 26632 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:15:56.527714 26632 rpc_server.cc:307] RPC server started. Bound to: 127.6.235.66:46641
I20260928 02:15:56.527753 26759 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.235.66:46641 every 8 connection(s)
I20260928 02:15:56.529151 26632 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790561432251112-7085-0/minicluster-data/ts-1/data/info.pb
I20260928 02:15:56.533814 7085 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTlaUuc/build/asan/bin/kudu as pid 26632
I20260928 02:15:56.534008 7085 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790561432251112-7085-0/minicluster-data/ts-1/wal/instance
I20260928 02:15:56.534559 7085 external_mini_cluster.cc:1428] Setting key fb140f29e0450f46a01c1d7e0c1b61f8
Sep 28 02:15:56 dist-test-slave-86gv krb5kdc[26392](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561756, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.235.66@KRBTEST.COM for kudu/127.6.235.126@KRBTEST.COM
I20260928 02:15:56.554692 26762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:15:56.531792 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.235.66:47273 (local address 127.6.235.126:36891)
0928 02:15:56.532256 (+ 464us) server_negotiation.cc:206] Beginning negotiation
0928 02:15:56.532260 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:15:56.533355 (+ 1095us) server_negotiation.cc:407] Connection header received
0928 02:15:56.535965 (+ 2610us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:15:56.535970 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:15:56.536050 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:15:56.536172 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:15:56.538254 (+ 2082us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:15:56.539042 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:15:56.541660 (+ 2618us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:15:56.542149 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:15:56.542258 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:15:56.546898 (+ 4640us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:15:56.546903 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:15:56.546906 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:15:56.546941 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:15:56.550828 (+ 3887us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:15:56.551656 (+ 828us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:15:56.551664 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:15:56.551670 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:15:56.551844 (+ 174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:15:56.552327 (+ 483us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:15:56.552334 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:15:56.552340 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:15:56.552729 (+ 389us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:15:56.552925 (+ 196us) server_negotiation.cc:1035] Waiting for connection context
0928 02:15:56.553880 (+ 955us) server_negotiation.cc:299] Negotiation successful
0928 02:15:56.554177 (+ 297us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":358,"thread_start_us":248,"threads_started":1}
I20260928 02:15:56.556020 26763 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:15:56.532421 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.235.126:36891 (local address 127.6.235.66:47273)
0928 02:15:56.533170 (+ 749us) negotiation.cc:107] Waiting for socket to connect
0928 02:15:56.533212 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0928 02:15:56.535659 (+ 2447us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:15:56.536383 (+ 724us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:15:56.536392 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:15:56.537187 (+ 795us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:15:56.538020 (+ 833us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:15:56.538089 (+ 69us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:15:56.539224 (+ 1135us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:15:56.539233 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:15:56.541490 (+ 2257us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:15:56.541508 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:15:56.542402 (+ 894us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:15:56.542411 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:15:56.542744 (+ 333us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:15:56.543549 (+ 805us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:15:56.543589 (+ 40us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:15:56.546619 (+ 3030us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:15:56.551001 (+ 4382us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:15:56.551009 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:15:56.551026 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:15:56.551495 (+ 469us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:15:56.551962 (+ 467us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:15:56.551965 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:15:56.551967 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:15:56.552188 (+ 221us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:15:56.552983 (+ 795us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:15:56.552990 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:15:56.553437 (+ 447us) client_negotiation.cc:769] Sending connection context
0928 02:15:56.553754 (+ 317us) client_negotiation.cc:240] Negotiation successful
0928 02:15:56.554302 (+ 548us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":486,"thread_start_us":258,"threads_started":1}
I20260928 02:15:56.557848 26760 heartbeater.cc:347] Connected to a master server at 127.6.235.126:36891
I20260928 02:15:56.558264 26760 heartbeater.cc:464] Registering TS with master...
I20260928 02:15:56.559105 26760 heartbeater.cc:511] Master 127.6.235.126:36891 requested a full tablet report, sending...
I20260928 02:15:56.560812 26437 ts_manager.cc:194] Registered new tserver with Master: 47cf21fa9bd045de8d3f13e19e60751a (127.6.235.66:46641)
I20260928 02:15:56.562136 26437 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.235.66@KRBTEST.COM'} at 127.6.235.66:47273
WARNING: no policy specified for kudu/127.6.235.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.235.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.235.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790561432251112-7085-0/krb5kdc/kudu_127.6.235.67.keytab.
Entry for principal kudu/127.6.235.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790561432251112-7085-0/krb5kdc/kudu_127.6.235.67.keytab.
I20260928 02:15:56.600579 7085 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTlaUuc/build/asan/bin/kudu
/tmp/dist-test-taskTlaUuc/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790561432251112-7085-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790561432251112-7085-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790561432251112-7085-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790561432251112-7085-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.235.67:0
--local_ip_for_outbound_sockets=127.6.235.67
--webserver_interface=127.6.235.67
--webserver_port=0
--tserver_master_addrs=127.6.235.126:36891
--keytab_file=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790561432251112-7085-0/krb5kdc/kudu_127.6.235.67.keytab
--principal=kudu/127.6.235.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.235.84:36435
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790561432251112-7085-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790561432251112-7085-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790561432251112-7085-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:15:56.949434 26766 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:15:56.949715 26766 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:15:56.949877 26766 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:15:56.961534 26766 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:15:56.961658 26766 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:15:56.961850 26766 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.235.67
I20260928 02:15:56.972095 26766 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.235.84:36435
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790561432251112-7085-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790561432251112-7085-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790561432251112-7085-0/krb5kdc/kudu_127.6.235.67.keytab
--principal=kudu/127.6.235.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.235.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790561432251112-7085-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.6.235.67
--webserver_port=0
--tserver_master_addrs=127.6.235.126:36891
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.235.67
--log_dir=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790561432251112-7085-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 00:51:59 UTC on ccycloud.quasar-axgkwn.root.comops.site
build id 4676464
ASAN enabled
I20260928 02:15:56.973572 26766 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:15:56.975092 26766 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260928 02:15:56.986456 26626 heartbeater.cc:503] Master 127.6.235.126:36891 was elected leader, sending a full tablet report...
W20260928 02:15:56.989550 26772 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:15:56.989471 26774 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:15:56.989521 26771 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:15:56.990590 26766 server_base.cc:1034] running on GCE node
I20260928 02:15:56.991358 26766 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:15:56.992645 26766 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:15:56.993885 26766 hybrid_clock.cc:648] HybridClock initialized: now 1790561756993786 us; error 99 us; skew 500 ppm
I20260928 02:15:56.994297 26766 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 02:15:56 dist-test-slave-86gv krb5kdc[26392](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561756, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.235.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:15:56.997778 26766 init.cc:377] Logged in from keytab as kudu/127.6.235.67@KRBTEST.COM (short username kudu)
I20260928 02:15:56.999449 26766 webserver.cc:460] Webserver started at http://127.6.235.67:43001/ using document root <none> and password file <none>
I20260928 02:15:57.000185 26766 fs_manager.cc:359] Metadata directory not provided
I20260928 02:15:57.000300 26766 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:15:57.000586 26766 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:15:57.003839 26766 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790561432251112-7085-0/minicluster-data/ts-2/data/instance:
uuid: "6ec7657c460e430da93948147ccbbff8"
format_stamp: "Formatted at 2026-09-28 02:15:56 on dist-test-slave-86gv"
server_key: "2726debd56bdfde3e24c7968917dd52b"
server_key_iv: "e429e02f755a82aa7a24348f4f6b1ee6"
server_key_version: "encryptionkey@0"
I20260928 02:15:57.004662 26766 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790561432251112-7085-0/minicluster-data/ts-2/wal/instance:
uuid: "6ec7657c460e430da93948147ccbbff8"
format_stamp: "Formatted at 2026-09-28 02:15:56 on dist-test-slave-86gv"
server_key: "2726debd56bdfde3e24c7968917dd52b"
server_key_iv: "e429e02f755a82aa7a24348f4f6b1ee6"
server_key_version: "encryptionkey@0"
I20260928 02:15:57.010669 26766 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.004s sys 0.000s
I20260928 02:15:57.014730 26781 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:15:57.016340 26766 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260928 02:15:57.016528 26766 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790561432251112-7085-0/minicluster-data/ts-2/data,/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790561432251112-7085-0/minicluster-data/ts-2/wal
uuid: "6ec7657c460e430da93948147ccbbff8"
format_stamp: "Formatted at 2026-09-28 02:15:56 on dist-test-slave-86gv"
server_key: "2726debd56bdfde3e24c7968917dd52b"
server_key_iv: "e429e02f755a82aa7a24348f4f6b1ee6"
server_key_version: "encryptionkey@0"
I20260928 02:15:57.016772 26766 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790561432251112-7085-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790561432251112-7085-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790561432251112-7085-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:15:57.037129 26766 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:15:57.042119 26766 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:15:57.042482 26766 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:15:57.043565 26766 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:15:57.045115 26766 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:15:57.045195 26766 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:15:57.045359 26766 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:15:57.045416 26766 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:15:57.081293 26766 rpc_server.cc:307] RPC server started. Bound to: 127.6.235.67:43229
I20260928 02:15:57.081341 26893 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.235.67:43229 every 8 connection(s)
I20260928 02:15:57.082789 26766 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790561432251112-7085-0/minicluster-data/ts-2/data/info.pb
I20260928 02:15:57.087895 7085 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTlaUuc/build/asan/bin/kudu as pid 26766
I20260928 02:15:57.088089 7085 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790561432251112-7085-0/minicluster-data/ts-2/wal/instance
I20260928 02:15:57.088562 7085 external_mini_cluster.cc:1428] Setting key 0d0cf4977c97d7c9c8665342bb57ff01
Sep 28 02:15:57 dist-test-slave-86gv krb5kdc[26392](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561756, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.235.67@KRBTEST.COM for kudu/127.6.235.126@KRBTEST.COM
I20260928 02:15:57.104365 26896 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:15:57.085637 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.235.67:49965 (local address 127.6.235.126:36891)
0928 02:15:57.086114 (+ 477us) server_negotiation.cc:206] Beginning negotiation
0928 02:15:57.086119 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 02:15:57.087244 (+ 1125us) server_negotiation.cc:407] Connection header received
0928 02:15:57.089496 (+ 2252us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:15:57.089502 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:15:57.089635 (+ 133us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:15:57.089759 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:15:57.091525 (+ 1766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:15:57.092353 (+ 828us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:15:57.094037 (+ 1684us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:15:57.094432 (+ 395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:15:57.094577 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:15:57.098341 (+ 3764us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:15:57.098347 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:15:57.098350 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:15:57.098393 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:15:57.101160 (+ 2767us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:15:57.101939 (+ 779us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:15:57.101951 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:15:57.101954 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:15:57.102078 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:15:57.102534 (+ 456us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:15:57.102543 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:15:57.102545 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:15:57.102837 (+ 292us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:15:57.103007 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0928 02:15:57.103726 (+ 719us) server_negotiation.cc:299] Negotiation successful
0928 02:15:57.103960 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":360,"thread_start_us":240,"threads_started":1}
I20260928 02:15:57.105432 26897 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:15:57.086322 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.235.126:36891 (local address 127.6.235.67:49965)
0928 02:15:57.087076 (+ 754us) negotiation.cc:107] Waiting for socket to connect
0928 02:15:57.087122 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0928 02:15:57.089143 (+ 2021us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:15:57.090063 (+ 920us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:15:57.090075 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:15:57.090730 (+ 655us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:15:57.091344 (+ 614us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:15:57.091362 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:15:57.092520 (+ 1158us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:15:57.092525 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:15:57.093877 (+ 1352us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:15:57.093891 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:15:57.094677 (+ 786us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:15:57.094686 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:15:57.094937 (+ 251us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:15:57.095649 (+ 712us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:15:57.095676 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:15:57.098108 (+ 2432us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:15:57.101315 (+ 3207us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:15:57.101326 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:15:57.101345 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:15:57.101800 (+ 455us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:15:57.102205 (+ 405us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:15:57.102210 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:15:57.102214 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:15:57.102408 (+ 194us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:15:57.102971 (+ 563us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:15:57.102980 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:15:57.103363 (+ 383us) client_negotiation.cc:769] Sending connection context
0928 02:15:57.103637 (+ 274us) client_negotiation.cc:240] Negotiation successful
0928 02:15:57.104222 (+ 585us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":500,"thread_start_us":315,"threads_started":1}
I20260928 02:15:57.107069 26894 heartbeater.cc:347] Connected to a master server at 127.6.235.126:36891
I20260928 02:15:57.107353 26894 heartbeater.cc:464] Registering TS with master...
I20260928 02:15:57.108093 26894 heartbeater.cc:511] Master 127.6.235.126:36891 requested a full tablet report, sending...
I20260928 02:15:57.109640 26437 ts_manager.cc:194] Registered new tserver with Master: 6ec7657c460e430da93948147ccbbff8 (127.6.235.67:43229)
I20260928 02:15:57.110446 26437 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.235.67@KRBTEST.COM'} at 127.6.235.67:49965
I20260928 02:15:57.116940 7085 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 02:15:57.134423 26896 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:15:57.121360 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47688 (local address 127.6.235.126:36891)
0928 02:15:57.121642 (+ 282us) server_negotiation.cc:206] Beginning negotiation
0928 02:15:57.121677 (+ 35us) server_negotiation.cc:399] Waiting for connection header
0928 02:15:57.122091 (+ 414us) server_negotiation.cc:407] Connection header received
0928 02:15:57.122239 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:15:57.122243 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:15:57.122318 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:15:57.122402 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:15:57.123979 (+ 1577us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:15:57.124721 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:15:57.126165 (+ 1444us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:15:57.126615 (+ 450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:15:57.126722 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:15:57.129216 (+ 2494us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:15:57.129221 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:15:57.129224 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:15:57.129259 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:15:57.131786 (+ 2527us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:15:57.132475 (+ 689us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:15:57.132478 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:15:57.132481 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:15:57.132617 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:15:57.132997 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:15:57.133001 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:15:57.133003 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:15:57.133335 (+ 332us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:15:57.133526 (+ 191us) server_negotiation.cc:1035] Waiting for connection context
0928 02:15:57.133862 (+ 336us) server_negotiation.cc:299] Negotiation successful
0928 02:15:57.134085 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260928 02:15:57.138893 26437 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47688:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 02:15:57.146160 26437 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 02:15:57.185855 26908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:15:57.161064 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.235.67:43229 (local address 127.0.0.1:33348)
0928 02:15:57.173185 (+ 12121us) negotiation.cc:107] Waiting for socket to connect
0928 02:15:57.173241 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0928 02:15:57.173490 (+ 249us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:15:57.175592 (+ 2102us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:15:57.175604 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:15:57.175653 (+ 49us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:15:57.176133 (+ 480us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:15:57.176146 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:15:57.178347 (+ 2201us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:15:57.178356 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:15:57.181468 (+ 3112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:15:57.181486 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:15:57.184639 (+ 3153us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:15:57.184650 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:15:57.184928 (+ 278us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:15:57.184996 (+ 68us) client_negotiation.cc:769] Sending connection context
0928 02:15:57.185140 (+ 144us) client_negotiation.cc:240] Negotiation successful
0928 02:15:57.185361 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11790,"mutex_wait_us":1,"thread_start_us":13552,"threads_started":1}
I20260928 02:15:57.187300 26910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:15:57.161692 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33348 (local address 127.6.235.67:43229)
0928 02:15:57.174489 (+ 12797us) server_negotiation.cc:206] Beginning negotiation
0928 02:15:57.174498 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 02:15:57.174538 (+ 40us) server_negotiation.cc:407] Connection header received
0928 02:15:57.174662 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:15:57.174674 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:15:57.174990 (+ 316us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:15:57.175235 (+ 245us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:15:57.176350 (+ 1115us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:15:57.178142 (+ 1792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:15:57.181665 (+ 3523us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:15:57.183904 (+ 2239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:15:57.184064 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:15:57.184442 (+ 378us) server_negotiation.cc:1035] Waiting for connection context
0928 02:15:57.186525 (+ 2083us) server_negotiation.cc:299] Negotiation successful
0928 02:15:57.186802 (+ 277us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12581,"thread_start_us":12448,"threads_started":1}
I20260928 02:15:57.188468 26907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:15:57.161209 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.235.66:46641 (local address 127.0.0.1:54304)
0928 02:15:57.174979 (+ 13770us) negotiation.cc:107] Waiting for socket to connect
0928 02:15:57.175001 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0928 02:15:57.175137 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:15:57.176338 (+ 1201us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:15:57.176346 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:15:57.176383 (+ 37us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:15:57.177026 (+ 643us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:15:57.177042 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:15:57.179610 (+ 2568us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:15:57.179623 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:15:57.184240 (+ 4617us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:15:57.184267 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:15:57.187370 (+ 3103us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:15:57.187378 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:15:57.187704 (+ 326us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:15:57.187768 (+ 64us) client_negotiation.cc:769] Sending connection context
0928 02:15:57.187877 (+ 109us) client_negotiation.cc:240] Negotiation successful
0928 02:15:57.188042 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13608,"mutex_wait_us":57,"thread_start_us":11480,"threads_started":1}
I20260928 02:15:57.189420 26909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:15:57.161576 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54304 (local address 127.6.235.66:46641)
0928 02:15:57.172608 (+ 11032us) server_negotiation.cc:206] Beginning negotiation
0928 02:15:57.172624 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0928 02:15:57.175283 (+ 2659us) server_negotiation.cc:407] Connection header received
0928 02:15:57.175413 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:15:57.175425 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:15:57.175736 (+ 311us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:15:57.175999 (+ 263us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:15:57.177296 (+ 1297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:15:57.179382 (+ 2086us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:15:57.184525 (+ 5143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:15:57.186654 (+ 2129us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:15:57.186813 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:15:57.187111 (+ 298us) server_negotiation.cc:1035] Waiting for connection context
0928 02:15:57.188858 (+ 1747us) server_negotiation.cc:299] Negotiation successful
0928 02:15:57.189106 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10736,"thread_start_us":10576,"threads_started":1}
I20260928 02:15:57.192961 26829 tablet_service.cc:1467] Processing CreateTablet for tablet 3516a5d1155140c2bd20788c10337051 (DEFAULT_TABLE table=test-table [id=0d602357f9e742eab7fcefe60a256066]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:15:57.193565 26695 tablet_service.cc:1467] Processing CreateTablet for tablet 3516a5d1155140c2bd20788c10337051 (DEFAULT_TABLE table=test-table [id=0d602357f9e742eab7fcefe60a256066]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:15:57.194718 26829 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3516a5d1155140c2bd20788c10337051. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:15:57.195112 26695 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3516a5d1155140c2bd20788c10337051. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:15:57.196265 26908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:15:57.175162 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.235.65:43845 (local address 127.0.0.1:55878)
0928 02:15:57.186113 (+ 10951us) negotiation.cc:107] Waiting for socket to connect
0928 02:15:57.186135 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0928 02:15:57.186259 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:15:57.189430 (+ 3171us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:15:57.189437 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:15:57.189536 (+ 99us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:15:57.189726 (+ 190us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:15:57.189739 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:15:57.192336 (+ 2597us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:15:57.192345 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:15:57.193838 (+ 1493us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:15:57.193863 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:15:57.195397 (+ 1534us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:15:57.195403 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:15:57.195606 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:15:57.195651 (+ 45us) client_negotiation.cc:769] Sending connection context
0928 02:15:57.195776 (+ 125us) client_negotiation.cc:240] Negotiation successful
0928 02:15:57.195940 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10844,"thread_start_us":12683,"threads_started":1}
I20260928 02:15:57.196327 26912 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:15:57.175428 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55878 (local address 127.6.235.65:43845)
0928 02:15:57.188610 (+ 13182us) server_negotiation.cc:206] Beginning negotiation
0928 02:15:57.188622 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0928 02:15:57.188656 (+ 34us) server_negotiation.cc:407] Connection header received
0928 02:15:57.188758 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:15:57.188769 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:15:57.189092 (+ 323us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:15:57.189306 (+ 214us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:15:57.189901 (+ 595us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:15:57.192180 (+ 2279us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:15:57.194038 (+ 1858us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:15:57.195185 (+ 1147us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:15:57.195346 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:15:57.195601 (+ 255us) server_negotiation.cc:1035] Waiting for connection context
0928 02:15:57.195840 (+ 239us) server_negotiation.cc:299] Negotiation successful
0928 02:15:57.196020 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12993,"thread_start_us":12840,"threads_started":1}
I20260928 02:15:57.200351 26561 tablet_service.cc:1467] Processing CreateTablet for tablet 3516a5d1155140c2bd20788c10337051 (DEFAULT_TABLE table=test-table [id=0d602357f9e742eab7fcefe60a256066]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:15:57.201675 26561 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3516a5d1155140c2bd20788c10337051. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:15:57.210527 26913 tablet_bootstrap.cc:492] T 3516a5d1155140c2bd20788c10337051 P 6ec7657c460e430da93948147ccbbff8: Bootstrap starting.
I20260928 02:15:57.211429 26914 tablet_bootstrap.cc:492] T 3516a5d1155140c2bd20788c10337051 P 47cf21fa9bd045de8d3f13e19e60751a: Bootstrap starting.
I20260928 02:15:57.215328 26914 tablet_bootstrap.cc:654] T 3516a5d1155140c2bd20788c10337051 P 47cf21fa9bd045de8d3f13e19e60751a: Neither blocks nor log segments found. Creating new log.
I20260928 02:15:57.215791 26915 tablet_bootstrap.cc:492] T 3516a5d1155140c2bd20788c10337051 P 05bc56491dbc468fbab24c4367a76967: Bootstrap starting.
I20260928 02:15:57.216495 26913 tablet_bootstrap.cc:654] T 3516a5d1155140c2bd20788c10337051 P 6ec7657c460e430da93948147ccbbff8: Neither blocks nor log segments found. Creating new log.
I20260928 02:15:57.217038 26914 log.cc:824] T 3516a5d1155140c2bd20788c10337051 P 47cf21fa9bd045de8d3f13e19e60751a: Log is configured to *not* fsync() on all Append() calls
I20260928 02:15:57.217841 26913 log.cc:824] T 3516a5d1155140c2bd20788c10337051 P 6ec7657c460e430da93948147ccbbff8: Log is configured to *not* fsync() on all Append() calls
I20260928 02:15:57.220153 26915 tablet_bootstrap.cc:654] T 3516a5d1155140c2bd20788c10337051 P 05bc56491dbc468fbab24c4367a76967: Neither blocks nor log segments found. Creating new log.
I20260928 02:15:57.221911 26915 log.cc:824] T 3516a5d1155140c2bd20788c10337051 P 05bc56491dbc468fbab24c4367a76967: Log is configured to *not* fsync() on all Append() calls
I20260928 02:15:57.224876 26913 tablet_bootstrap.cc:492] T 3516a5d1155140c2bd20788c10337051 P 6ec7657c460e430da93948147ccbbff8: No bootstrap required, opened a new log
I20260928 02:15:57.225286 26913 ts_tablet_manager.cc:1397] T 3516a5d1155140c2bd20788c10337051 P 6ec7657c460e430da93948147ccbbff8: Time spent bootstrapping tablet: real 0.015s user 0.007s sys 0.004s
I20260928 02:15:57.225397 26915 tablet_bootstrap.cc:492] T 3516a5d1155140c2bd20788c10337051 P 05bc56491dbc468fbab24c4367a76967: No bootstrap required, opened a new log
I20260928 02:15:57.225581 26914 tablet_bootstrap.cc:492] T 3516a5d1155140c2bd20788c10337051 P 47cf21fa9bd045de8d3f13e19e60751a: No bootstrap required, opened a new log
I20260928 02:15:57.225695 26915 ts_tablet_manager.cc:1397] T 3516a5d1155140c2bd20788c10337051 P 05bc56491dbc468fbab24c4367a76967: Time spent bootstrapping tablet: real 0.010s user 0.009s sys 0.000s
I20260928 02:15:57.225878 26914 ts_tablet_manager.cc:1397] T 3516a5d1155140c2bd20788c10337051 P 47cf21fa9bd045de8d3f13e19e60751a: Time spent bootstrapping tablet: real 0.015s user 0.013s sys 0.000s
I20260928 02:15:57.233911 26914 raft_consensus.cc:348] T 3516a5d1155140c2bd20788c10337051 P 47cf21fa9bd045de8d3f13e19e60751a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ec7657c460e430da93948147ccbbff8" member_type: VOTER last_known_addr { host: "127.6.235.67" port: 43229 } } peers { permanent_uuid: "47cf21fa9bd045de8d3f13e19e60751a" member_type: VOTER last_known_addr { host: "127.6.235.66" port: 46641 } } peers { permanent_uuid: "05bc56491dbc468fbab24c4367a76967" member_type: VOTER last_known_addr { host: "127.6.235.65" port: 43845 } }
I20260928 02:15:57.234460 26914 raft_consensus.cc:374] T 3516a5d1155140c2bd20788c10337051 P 47cf21fa9bd045de8d3f13e19e60751a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:15:57.234565 26914 raft_consensus.cc:729] T 3516a5d1155140c2bd20788c10337051 P 47cf21fa9bd045de8d3f13e19e60751a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 47cf21fa9bd045de8d3f13e19e60751a, State: Initialized, Role: FOLLOWER
I20260928 02:15:57.235262 26914 consensus_queue.cc:260] T 3516a5d1155140c2bd20788c10337051 P 47cf21fa9bd045de8d3f13e19e60751a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ec7657c460e430da93948147ccbbff8" member_type: VOTER last_known_addr { host: "127.6.235.67" port: 43229 } } peers { permanent_uuid: "47cf21fa9bd045de8d3f13e19e60751a" member_type: VOTER last_known_addr { host: "127.6.235.66" port: 46641 } } peers { permanent_uuid: "05bc56491dbc468fbab24c4367a76967" member_type: VOTER last_known_addr { host: "127.6.235.65" port: 43845 } }
I20260928 02:15:57.236487 26760 heartbeater.cc:503] Master 127.6.235.126:36891 was elected leader, sending a full tablet report...
I20260928 02:15:57.237323 26914 ts_tablet_manager.cc:1428] T 3516a5d1155140c2bd20788c10337051 P 47cf21fa9bd045de8d3f13e19e60751a: Time spent starting tablet: real 0.011s user 0.006s sys 0.005s
I20260928 02:15:57.237432 26913 raft_consensus.cc:348] T 3516a5d1155140c2bd20788c10337051 P 6ec7657c460e430da93948147ccbbff8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ec7657c460e430da93948147ccbbff8" member_type: VOTER last_known_addr { host: "127.6.235.67" port: 43229 } } peers { permanent_uuid: "47cf21fa9bd045de8d3f13e19e60751a" member_type: VOTER last_known_addr { host: "127.6.235.66" port: 46641 } } peers { permanent_uuid: "05bc56491dbc468fbab24c4367a76967" member_type: VOTER last_known_addr { host: "127.6.235.65" port: 43845 } }
I20260928 02:15:57.237432 26915 raft_consensus.cc:348] T 3516a5d1155140c2bd20788c10337051 P 05bc56491dbc468fbab24c4367a76967 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ec7657c460e430da93948147ccbbff8" member_type: VOTER last_known_addr { host: "127.6.235.67" port: 43229 } } peers { permanent_uuid: "47cf21fa9bd045de8d3f13e19e60751a" member_type: VOTER last_known_addr { host: "127.6.235.66" port: 46641 } } peers { permanent_uuid: "05bc56491dbc468fbab24c4367a76967" member_type: VOTER last_known_addr { host: "127.6.235.65" port: 43845 } }
I20260928 02:15:57.237900 26915 raft_consensus.cc:374] T 3516a5d1155140c2bd20788c10337051 P 05bc56491dbc468fbab24c4367a76967 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:15:57.237900 26913 raft_consensus.cc:374] T 3516a5d1155140c2bd20788c10337051 P 6ec7657c460e430da93948147ccbbff8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:15:57.237998 26913 raft_consensus.cc:729] T 3516a5d1155140c2bd20788c10337051 P 6ec7657c460e430da93948147ccbbff8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6ec7657c460e430da93948147ccbbff8, State: Initialized, Role: FOLLOWER
I20260928 02:15:57.238013 26915 raft_consensus.cc:729] T 3516a5d1155140c2bd20788c10337051 P 05bc56491dbc468fbab24c4367a76967 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 05bc56491dbc468fbab24c4367a76967, State: Initialized, Role: FOLLOWER
I20260928 02:15:57.238781 26915 consensus_queue.cc:260] T 3516a5d1155140c2bd20788c10337051 P 05bc56491dbc468fbab24c4367a76967 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ec7657c460e430da93948147ccbbff8" member_type: VOTER last_known_addr { host: "127.6.235.67" port: 43229 } } peers { permanent_uuid: "47cf21fa9bd045de8d3f13e19e60751a" member_type: VOTER last_known_addr { host: "127.6.235.66" port: 46641 } } peers { permanent_uuid: "05bc56491dbc468fbab24c4367a76967" member_type: VOTER last_known_addr { host: "127.6.235.65" port: 43845 } }
I20260928 02:15:57.239859 26913 consensus_queue.cc:260] T 3516a5d1155140c2bd20788c10337051 P 6ec7657c460e430da93948147ccbbff8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ec7657c460e430da93948147ccbbff8" member_type: VOTER last_known_addr { host: "127.6.235.67" port: 43229 } } peers { permanent_uuid: "47cf21fa9bd045de8d3f13e19e60751a" member_type: VOTER last_known_addr { host: "127.6.235.66" port: 46641 } } peers { permanent_uuid: "05bc56491dbc468fbab24c4367a76967" member_type: VOTER last_known_addr { host: "127.6.235.65" port: 43845 } }
I20260928 02:15:57.242060 26894 heartbeater.cc:503] Master 127.6.235.126:36891 was elected leader, sending a full tablet report...
I20260928 02:15:57.242831 26913 ts_tablet_manager.cc:1428] T 3516a5d1155140c2bd20788c10337051 P 6ec7657c460e430da93948147ccbbff8: Time spent starting tablet: real 0.017s user 0.013s sys 0.002s
I20260928 02:15:57.244663 26915 ts_tablet_manager.cc:1428] T 3516a5d1155140c2bd20788c10337051 P 05bc56491dbc468fbab24c4367a76967: Time spent starting tablet: real 0.019s user 0.014s sys 0.004s
I20260928 02:15:57.278035 26920 raft_consensus.cc:482] T 3516a5d1155140c2bd20788c10337051 P 05bc56491dbc468fbab24c4367a76967 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 02:15:57.278379 26920 raft_consensus.cc:504] T 3516a5d1155140c2bd20788c10337051 P 05bc56491dbc468fbab24c4367a76967 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ec7657c460e430da93948147ccbbff8" member_type: VOTER last_known_addr { host: "127.6.235.67" port: 43229 } } peers { permanent_uuid: "47cf21fa9bd045de8d3f13e19e60751a" member_type: VOTER last_known_addr { host: "127.6.235.66" port: 46641 } } peers { permanent_uuid: "05bc56491dbc468fbab24c4367a76967" member_type: VOTER last_known_addr { host: "127.6.235.65" port: 43845 } }
I20260928 02:15:57.280296 26920 leader_election.cc:288] T 3516a5d1155140c2bd20788c10337051 P 05bc56491dbc468fbab24c4367a76967 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6ec7657c460e430da93948147ccbbff8 (127.6.235.67:43229), 47cf21fa9bd045de8d3f13e19e60751a (127.6.235.66:46641)
W20260928 02:15:57.281579 26761 tablet.cc:2369] T 3516a5d1155140c2bd20788c10337051 P 47cf21fa9bd045de8d3f13e19e60751a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 02:15:57.288964 26910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:15:57.280996 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.235.65:42085 (local address 127.6.235.67:43229)
0928 02:15:57.281248 (+ 252us) server_negotiation.cc:206] Beginning negotiation
0928 02:15:57.281254 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 02:15:57.283117 (+ 1863us) server_negotiation.cc:407] Connection header received
0928 02:15:57.283217 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:15:57.283226 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:15:57.283309 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:15:57.283393 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:15:57.284061 (+ 668us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:15:57.285088 (+ 1027us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:15:57.287000 (+ 1912us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:15:57.287939 (+ 939us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:15:57.288084 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:15:57.288141 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0928 02:15:57.288535 (+ 394us) server_negotiation.cc:299] Negotiation successful
0928 02:15:57.288674 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260928 02:15:57.289110 26923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:15:57.280694 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.235.67:43229 (local address 127.6.235.65:42085)
0928 02:15:57.282979 (+ 2285us) negotiation.cc:107] Waiting for socket to connect
0928 02:15:57.282994 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 02:15:57.283108 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:15:57.283429 (+ 321us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:15:57.283433 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:15:57.283457 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:15:57.283865 (+ 408us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:15:57.283878 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:15:57.285224 (+ 1346us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:15:57.285229 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:15:57.286816 (+ 1587us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:15:57.286832 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:15:57.288167 (+ 1335us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:15:57.288174 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:15:57.288347 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:15:57.288395 (+ 48us) client_negotiation.cc:769] Sending connection context
0928 02:15:57.288489 (+ 94us) client_negotiation.cc:240] Negotiation successful
0928 02:15:57.288645 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2120,"mutex_wait_us":23,"spinlock_wait_cycles":768,"thread_start_us":1801,"threads_started":1}
I20260928 02:15:57.289390 26909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:15:57.280993 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.235.65:34721 (local address 127.6.235.66:46641)
0928 02:15:57.281441 (+ 448us) server_negotiation.cc:206] Beginning negotiation
0928 02:15:57.281447 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 02:15:57.282904 (+ 1457us) server_negotiation.cc:407] Connection header received
0928 02:15:57.283271 (+ 367us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:15:57.283274 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:15:57.283335 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:15:57.283441 (+ 106us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:15:57.284301 (+ 860us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:15:57.285236 (+ 935us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:15:57.287403 (+ 2167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:15:57.288320 (+ 917us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:15:57.288497 (+ 177us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:15:57.288576 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0928 02:15:57.288977 (+ 401us) server_negotiation.cc:299] Negotiation successful
0928 02:15:57.289116 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":282}
I20260928 02:15:57.289831 26922 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:15:57.280640 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.235.66:46641 (local address 127.6.235.65:34721)
0928 02:15:57.282770 (+ 2130us) negotiation.cc:107] Waiting for socket to connect
0928 02:15:57.282790 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0928 02:15:57.283027 (+ 237us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:15:57.283500 (+ 473us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:15:57.283505 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:15:57.283632 (+ 127us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:15:57.284117 (+ 485us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:15:57.284133 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:15:57.285492 (+ 1359us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:15:57.285496 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:15:57.287229 (+ 1733us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:15:57.287243 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:15:57.288579 (+ 1336us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:15:57.288586 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:15:57.288763 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:15:57.288798 (+ 35us) client_negotiation.cc:769] Sending connection context
0928 02:15:57.288935 (+ 137us) client_negotiation.cc:240] Negotiation successful
0928 02:15:57.289441 (+ 506us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1975,"mutex_wait_us":28,"spinlock_wait_cycles":38144,"thread_start_us":987,"threads_started":1}
I20260928 02:15:57.290316 26849 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3516a5d1155140c2bd20788c10337051" candidate_uuid: "05bc56491dbc468fbab24c4367a76967" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6ec7657c460e430da93948147ccbbff8" is_pre_election: true
I20260928 02:15:57.290881 26849 raft_consensus.cc:2454] T 3516a5d1155140c2bd20788c10337051 P 6ec7657c460e430da93948147ccbbff8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 05bc56491dbc468fbab24c4367a76967 in term 0.
I20260928 02:15:57.290836 26715 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3516a5d1155140c2bd20788c10337051" candidate_uuid: "05bc56491dbc468fbab24c4367a76967" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "47cf21fa9bd045de8d3f13e19e60751a" is_pre_election: true
I20260928 02:15:57.291409 26715 raft_consensus.cc:2454] T 3516a5d1155140c2bd20788c10337051 P 47cf21fa9bd045de8d3f13e19e60751a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 05bc56491dbc468fbab24c4367a76967 in term 0.
I20260928 02:15:57.291648 26516 leader_election.cc:302] T 3516a5d1155140c2bd20788c10337051 P 05bc56491dbc468fbab24c4367a76967 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 05bc56491dbc468fbab24c4367a76967, 6ec7657c460e430da93948147ccbbff8; no voters:
I20260928 02:15:57.292059 26920 raft_consensus.cc:2781] T 3516a5d1155140c2bd20788c10337051 P 05bc56491dbc468fbab24c4367a76967 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 02:15:57.292207 26920 raft_consensus.cc:482] T 3516a5d1155140c2bd20788c10337051 P 05bc56491dbc468fbab24c4367a76967 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 02:15:57.292315 26920 raft_consensus.cc:3037] T 3516a5d1155140c2bd20788c10337051 P 05bc56491dbc468fbab24c4367a76967 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:15:57.294728 26920 raft_consensus.cc:504] T 3516a5d1155140c2bd20788c10337051 P 05bc56491dbc468fbab24c4367a76967 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ec7657c460e430da93948147ccbbff8" member_type: VOTER last_known_addr { host: "127.6.235.67" port: 43229 } } peers { permanent_uuid: "47cf21fa9bd045de8d3f13e19e60751a" member_type: VOTER last_known_addr { host: "127.6.235.66" port: 46641 } } peers { permanent_uuid: "05bc56491dbc468fbab24c4367a76967" member_type: VOTER last_known_addr { host: "127.6.235.65" port: 43845 } }
I20260928 02:15:57.295396 26920 leader_election.cc:288] T 3516a5d1155140c2bd20788c10337051 P 05bc56491dbc468fbab24c4367a76967 [CANDIDATE]: Term 1 election: Requested vote from peers 6ec7657c460e430da93948147ccbbff8 (127.6.235.67:43229), 47cf21fa9bd045de8d3f13e19e60751a (127.6.235.66:46641)
I20260928 02:15:57.295861 26849 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3516a5d1155140c2bd20788c10337051" candidate_uuid: "05bc56491dbc468fbab24c4367a76967" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6ec7657c460e430da93948147ccbbff8"
I20260928 02:15:57.295969 26715 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3516a5d1155140c2bd20788c10337051" candidate_uuid: "05bc56491dbc468fbab24c4367a76967" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "47cf21fa9bd045de8d3f13e19e60751a"
I20260928 02:15:57.296094 26849 raft_consensus.cc:3037] T 3516a5d1155140c2bd20788c10337051 P 6ec7657c460e430da93948147ccbbff8 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:15:57.296227 26715 raft_consensus.cc:3037] T 3516a5d1155140c2bd20788c10337051 P 47cf21fa9bd045de8d3f13e19e60751a [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:15:57.298476 26849 raft_consensus.cc:2454] T 3516a5d1155140c2bd20788c10337051 P 6ec7657c460e430da93948147ccbbff8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 05bc56491dbc468fbab24c4367a76967 in term 1.
I20260928 02:15:57.298482 26715 raft_consensus.cc:2454] T 3516a5d1155140c2bd20788c10337051 P 47cf21fa9bd045de8d3f13e19e60751a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 05bc56491dbc468fbab24c4367a76967 in term 1.
I20260928 02:15:57.298931 26516 leader_election.cc:302] T 3516a5d1155140c2bd20788c10337051 P 05bc56491dbc468fbab24c4367a76967 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 05bc56491dbc468fbab24c4367a76967, 6ec7657c460e430da93948147ccbbff8; no voters:
I20260928 02:15:57.299257 26920 raft_consensus.cc:2781] T 3516a5d1155140c2bd20788c10337051 P 05bc56491dbc468fbab24c4367a76967 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:15:57.299759 26920 raft_consensus.cc:686] T 3516a5d1155140c2bd20788c10337051 P 05bc56491dbc468fbab24c4367a76967 [term 1 LEADER]: Becoming Leader. State: Replica: 05bc56491dbc468fbab24c4367a76967, State: Running, Role: LEADER
I20260928 02:15:57.300256 26920 consensus_queue.cc:237] T 3516a5d1155140c2bd20788c10337051 P 05bc56491dbc468fbab24c4367a76967 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ec7657c460e430da93948147ccbbff8" member_type: VOTER last_known_addr { host: "127.6.235.67" port: 43229 } } peers { permanent_uuid: "47cf21fa9bd045de8d3f13e19e60751a" member_type: VOTER last_known_addr { host: "127.6.235.66" port: 46641 } } peers { permanent_uuid: "05bc56491dbc468fbab24c4367a76967" member_type: VOTER last_known_addr { host: "127.6.235.65" port: 43845 } }
I20260928 02:15:57.305094 26437 catalog_manager.cc:5495] T 3516a5d1155140c2bd20788c10337051 P 05bc56491dbc468fbab24c4367a76967 reported cstate change: term changed from 0 to 1, leader changed from <none> to 05bc56491dbc468fbab24c4367a76967 (127.6.235.65). New cstate: current_term: 1 leader_uuid: "05bc56491dbc468fbab24c4367a76967" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6ec7657c460e430da93948147ccbbff8" member_type: VOTER last_known_addr { host: "127.6.235.67" port: 43229 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "47cf21fa9bd045de8d3f13e19e60751a" member_type: VOTER last_known_addr { host: "127.6.235.66" port: 46641 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "05bc56491dbc468fbab24c4367a76967" member_type: VOTER last_known_addr { host: "127.6.235.65" port: 43845 } health_report { overall_health: HEALTHY } } }
W20260928 02:15:57.334632 26895 tablet.cc:2369] T 3516a5d1155140c2bd20788c10337051 P 6ec7657c460e430da93948147ccbbff8: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 02:15:57.346769 26912 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:15:57.341320 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55890 (local address 127.6.235.65:43845)
0928 02:15:57.341530 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0928 02:15:57.341535 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 02:15:57.341554 (+ 19us) server_negotiation.cc:407] Connection header received
0928 02:15:57.341649 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:15:57.341653 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:15:57.341745 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:15:57.341877 (+ 132us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 02:15:57.342270 (+ 393us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:15:57.343071 (+ 801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:15:57.344440 (+ 1369us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:15:57.344879 (+ 439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:15:57.345032 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:15:57.345577 (+ 545us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 02:15:57.346133 (+ 556us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 02:15:57.346245 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0928 02:15:57.346375 (+ 130us) server_negotiation.cc:299] Negotiation successful
0928 02:15:57.346448 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260928 02:15:57.357046 26715 raft_consensus.cc:1261] T 3516a5d1155140c2bd20788c10337051 P 47cf21fa9bd045de8d3f13e19e60751a [term 1 FOLLOWER]: Refusing update from remote peer 05bc56491dbc468fbab24c4367a76967: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 02:15:57.357046 26849 raft_consensus.cc:1261] T 3516a5d1155140c2bd20788c10337051 P 6ec7657c460e430da93948147ccbbff8 [term 1 FOLLOWER]: Refusing update from remote peer 05bc56491dbc468fbab24c4367a76967: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 02:15:57.358587 26924 consensus_queue.cc:1035] T 3516a5d1155140c2bd20788c10337051 P 05bc56491dbc468fbab24c4367a76967 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6ec7657c460e430da93948147ccbbff8" member_type: VOTER last_known_addr { host: "127.6.235.67" port: 43229 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 02:15:57.359021 26920 consensus_queue.cc:1035] T 3516a5d1155140c2bd20788c10337051 P 05bc56491dbc468fbab24c4367a76967 [LEADER]: Connected to new peer: Peer: permanent_uuid: "47cf21fa9bd045de8d3f13e19e60751a" member_type: VOTER last_known_addr { host: "127.6.235.66" port: 46641 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 02:15:57.376194 26927 mvcc.cc:205] Tried to move back new op lower bound from 7334140958121046016 to 7334140957906386944. Current Snapshot: MvccSnapshot[applied={T|T < 7334140958121046016 or (T in {7334140958121046016})}]
I20260928 02:15:57.386770 26437 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47688:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 02:15:57.387100 26437 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47688:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 02:15:57.391407 26437 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 68fabc3a8fd74b1ea6a30861cf2178e9: Sending DeleteTablet for 3 replicas of tablet 3516a5d1155140c2bd20788c10337051
I20260928 02:15:57.392802 26695 tablet_service.cc:1514] Processing DeleteTablet for tablet 3516a5d1155140c2bd20788c10337051 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:15:57 UTC) from {username='kudu', principal='kudu/127.6.235.126@KRBTEST.COM'} at 127.0.0.1:54304
I20260928 02:15:57.392804 26829 tablet_service.cc:1514] Processing DeleteTablet for tablet 3516a5d1155140c2bd20788c10337051 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:15:57 UTC) from {username='kudu', principal='kudu/127.6.235.126@KRBTEST.COM'} at 127.0.0.1:33348
I20260928 02:15:57.393909 7085 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskTlaUuc/build/asan/bin/kudu pid 26498
I20260928 02:15:57.394224 26561 tablet_service.cc:1514] Processing DeleteTablet for tablet 3516a5d1155140c2bd20788c10337051 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:15:57 UTC) from {username='kudu', principal='kudu/127.6.235.126@KRBTEST.COM'} at 127.0.0.1:55878
I20260928 02:15:57.403043 26938 tablet_replica.cc:330] T 3516a5d1155140c2bd20788c10337051 P 05bc56491dbc468fbab24c4367a76967: stopping tablet replica
I20260928 02:15:57.403672 26938 raft_consensus.cc:2229] T 3516a5d1155140c2bd20788c10337051 P 05bc56491dbc468fbab24c4367a76967 [term 1 LEADER]: Raft consensus shutting down.
I20260928 02:15:57.404574 26938 raft_consensus.cc:2258] T 3516a5d1155140c2bd20788c10337051 P 05bc56491dbc468fbab24c4367a76967 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 02:15:57.406493 26936 tablet_replica.cc:330] T 3516a5d1155140c2bd20788c10337051 P 47cf21fa9bd045de8d3f13e19e60751a: stopping tablet replica
I20260928 02:15:57.407014 26936 raft_consensus.cc:2229] T 3516a5d1155140c2bd20788c10337051 P 47cf21fa9bd045de8d3f13e19e60751a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 02:15:57.408257 26936 raft_consensus.cc:2258] T 3516a5d1155140c2bd20788c10337051 P 47cf21fa9bd045de8d3f13e19e60751a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 02:15:57.409462 26938 ts_tablet_manager.cc:1905] T 3516a5d1155140c2bd20788c10337051 P 05bc56491dbc468fbab24c4367a76967: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 02:15:57.410715 26936 ts_tablet_manager.cc:1905] T 3516a5d1155140c2bd20788c10337051 P 47cf21fa9bd045de8d3f13e19e60751a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 02:15:57.411464 26937 tablet_replica.cc:330] T 3516a5d1155140c2bd20788c10337051 P 6ec7657c460e430da93948147ccbbff8: stopping tablet replica
I20260928 02:15:57.412056 26937 raft_consensus.cc:2229] T 3516a5d1155140c2bd20788c10337051 P 6ec7657c460e430da93948147ccbbff8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 02:15:57.413460 26937 raft_consensus.cc:2258] T 3516a5d1155140c2bd20788c10337051 P 6ec7657c460e430da93948147ccbbff8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 02:15:57.418221 26938 ts_tablet_manager.cc:1918] T 3516a5d1155140c2bd20788c10337051 P 05bc56491dbc468fbab24c4367a76967: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 02:15:57.418212 26936 ts_tablet_manager.cc:1918] T 3516a5d1155140c2bd20788c10337051 P 47cf21fa9bd045de8d3f13e19e60751a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 02:15:57.418432 26936 log.cc:1196] T 3516a5d1155140c2bd20788c10337051 P 47cf21fa9bd045de8d3f13e19e60751a: Deleting WAL directory at /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790561432251112-7085-0/minicluster-data/ts-1/wal/wals/3516a5d1155140c2bd20788c10337051
I20260928 02:15:57.418421 26938 log.cc:1196] T 3516a5d1155140c2bd20788c10337051 P 05bc56491dbc468fbab24c4367a76967: Deleting WAL directory at /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790561432251112-7085-0/minicluster-data/ts-0/wal/wals/3516a5d1155140c2bd20788c10337051
I20260928 02:15:57.419015 26938 ts_tablet_manager.cc:1939] T 3516a5d1155140c2bd20788c10337051 P 05bc56491dbc468fbab24c4367a76967: Deleting consensus metadata
I20260928 02:15:57.419013 26936 ts_tablet_manager.cc:1939] T 3516a5d1155140c2bd20788c10337051 P 47cf21fa9bd045de8d3f13e19e60751a: Deleting consensus metadata
I20260928 02:15:57.421072 26422 catalog_manager.cc:4841] TS 47cf21fa9bd045de8d3f13e19e60751a (127.6.235.66:46641): tablet 3516a5d1155140c2bd20788c10337051 (table test-table [id=0d602357f9e742eab7fcefe60a256066]) successfully deleted
I20260928 02:15:57.421343 26937 ts_tablet_manager.cc:1905] T 3516a5d1155140c2bd20788c10337051 P 6ec7657c460e430da93948147ccbbff8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 02:15:57.421427 26423 catalog_manager.cc:4841] TS 05bc56491dbc468fbab24c4367a76967 (127.6.235.65:43845): tablet 3516a5d1155140c2bd20788c10337051 (table test-table [id=0d602357f9e742eab7fcefe60a256066]) successfully deleted
Sep 28 02:15:57 dist-test-slave-86gv krb5kdc[26392](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561754, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.235.65@KRBTEST.COM
I20260928 02:15:57.427310 26937 ts_tablet_manager.cc:1918] T 3516a5d1155140c2bd20788c10337051 P 6ec7657c460e430da93948147ccbbff8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 02:15:57.427523 26937 log.cc:1196] T 3516a5d1155140c2bd20788c10337051 P 6ec7657c460e430da93948147ccbbff8: Deleting WAL directory at /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790561432251112-7085-0/minicluster-data/ts-2/wal/wals/3516a5d1155140c2bd20788c10337051
I20260928 02:15:57.428136 26937 ts_tablet_manager.cc:1939] T 3516a5d1155140c2bd20788c10337051 P 6ec7657c460e430da93948147ccbbff8: Deleting consensus metadata
I20260928 02:15:57.430269 26423 catalog_manager.cc:4841] TS 6ec7657c460e430da93948147ccbbff8 (127.6.235.67:43229): tablet 3516a5d1155140c2bd20788c10337051 (table test-table [id=0d602357f9e742eab7fcefe60a256066]) successfully deleted
I20260928 02:15:57.435649 26912 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:15:57.397067 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55902 (local address 127.6.235.65:43845)
0928 02:15:57.397262 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0928 02:15:57.397268 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 02:15:57.407782 (+ 10514us) server_negotiation.cc:407] Connection header received
0928 02:15:57.408336 (+ 554us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:15:57.408344 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:15:57.408443 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:15:57.408546 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:15:57.413701 (+ 5155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:15:57.415118 (+ 1417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:15:57.420837 (+ 5719us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:15:57.421658 (+ 821us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:15:57.421787 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:15:57.428206 (+ 6419us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:15:57.428231 (+ 25us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:15:57.428252 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:15:57.428318 (+ 66us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:15:57.432303 (+ 3985us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:15:57.433304 (+ 1001us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:15:57.433312 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:15:57.433315 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:15:57.433489 (+ 174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:15:57.433952 (+ 463us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:15:57.433956 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:15:57.433958 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:15:57.434415 (+ 457us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:15:57.434705 (+ 290us) server_negotiation.cc:1035] Waiting for connection context
0928 02:15:57.435131 (+ 426us) server_negotiation.cc:299] Negotiation successful
0928 02:15:57.435333 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260928 02:15:57.436359 26621 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 02:15:57.540766 7085 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTlaUuc/build/asan/bin/kudu with pid 26498
I20260928 02:15:57.565253 7085 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskTlaUuc/build/asan/bin/kudu pid 26632
Sep 28 02:15:57 dist-test-slave-86gv krb5kdc[26392](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561754, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.235.66@KRBTEST.COM
I20260928 02:15:57.582106 26909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:15:57.565951 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54308 (local address 127.6.235.66:46641)
0928 02:15:57.566197 (+ 246us) server_negotiation.cc:206] Beginning negotiation
0928 02:15:57.566202 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 02:15:57.566221 (+ 19us) server_negotiation.cc:407] Connection header received
0928 02:15:57.566311 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:15:57.566314 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:15:57.566376 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:15:57.566459 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:15:57.567760 (+ 1301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:15:57.568492 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:15:57.569913 (+ 1421us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:15:57.570371 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:15:57.570494 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:15:57.575292 (+ 4798us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:15:57.575309 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:15:57.575326 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:15:57.575372 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:15:57.578929 (+ 3557us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:15:57.579926 (+ 997us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:15:57.579935 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:15:57.579938 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:15:57.580159 (+ 221us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:15:57.580555 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:15:57.580559 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:15:57.580561 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:15:57.581094 (+ 533us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:15:57.581338 (+ 244us) server_negotiation.cc:1035] Waiting for connection context
0928 02:15:57.581613 (+ 275us) server_negotiation.cc:299] Negotiation successful
0928 02:15:57.581799 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":125}
I20260928 02:15:57.582787 26755 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 02:15:57.675956 7085 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTlaUuc/build/asan/bin/kudu with pid 26632
I20260928 02:15:57.694497 7085 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskTlaUuc/build/asan/bin/kudu pid 26766
Sep 28 02:15:57 dist-test-slave-86gv krb5kdc[26392](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561754, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.235.67@KRBTEST.COM
I20260928 02:15:57.711087 26910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:15:57.695392 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33356 (local address 127.6.235.67:43229)
0928 02:15:57.695619 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0928 02:15:57.695625 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 02:15:57.695647 (+ 22us) server_negotiation.cc:407] Connection header received
0928 02:15:57.695769 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:15:57.695780 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:15:57.695856 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:15:57.695972 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:15:57.697497 (+ 1525us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:15:57.698223 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:15:57.699617 (+ 1394us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:15:57.700029 (+ 412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:15:57.700169 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:15:57.705005 (+ 4836us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:15:57.705030 (+ 25us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:15:57.705053 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:15:57.705095 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:15:57.708029 (+ 2934us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:15:57.708799 (+ 770us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:15:57.708807 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:15:57.708810 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:15:57.709020 (+ 210us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:15:57.709420 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:15:57.709426 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:15:57.709429 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:15:57.709799 (+ 370us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:15:57.710116 (+ 317us) server_negotiation.cc:1035] Waiting for connection context
0928 02:15:57.710432 (+ 316us) server_negotiation.cc:299] Negotiation successful
0928 02:15:57.710703 (+ 271us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260928 02:15:57.711896 26889 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 02:15:57.803134 7085 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTlaUuc/build/asan/bin/kudu with pid 26766
I20260928 02:15:57.822202 7085 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskTlaUuc/build/asan/bin/kudu pid 26405
I20260928 02:15:57.823371 26467 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 02:15:57.938665 7085 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTlaUuc/build/asan/bin/kudu with pid 26405
2026-09-28T02:15:57Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/4 (4570 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/5
2026-09-28T02:15:57Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T02:15:57Z Disabled control of system clock
I20260928 02:15:58.020280 7085 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTlaUuc/build/asan/bin/kudu
/tmp/dist-test-taskTlaUuc/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790561432251112-7085-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790561432251112-7085-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790561432251112-7085-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790561432251112-7085-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.235.126:32827
--webserver_interface=127.6.235.126
--webserver_port=0
--builtin_ntp_servers=127.6.235.84:42579
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.235.126:32827
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260928 02:15:58.372762 26947 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:15:58.373126 26947 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:15:58.373211 26947 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:15:58.380743 26947 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 02:15:58.380874 26947 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:15:58.381014 26947 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:15:58.381065 26947 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 02:15:58.381110 26947 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 02:15:58.390789 26947 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.235.84:42579
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790561432251112-7085-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790561432251112-7085-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.235.126:32827
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.235.126:32827
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790561432251112-7085-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.235.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790561432251112-7085-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 00:51:59 UTC on ccycloud.quasar-axgkwn.root.comops.site
build id 4676464
ASAN enabled
I20260928 02:15:58.392123 26947 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:15:58.393945 26947 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:15:58.406747 26952 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:15:58.406728 26953 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:15:58.407133 26955 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:15:58.407696 26947 server_base.cc:1034] running on GCE node
I20260928 02:15:58.408550 26947 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:15:58.410156 26947 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:15:58.411379 26947 hybrid_clock.cc:648] HybridClock initialized: now 1790561758411315 us; error 66 us; skew 500 ppm
I20260928 02:15:58.411803 26947 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260928 02:15:58.414445 26947 webserver.cc:460] Webserver started at http://127.6.235.126:40353/ using document root <none> and password file <none>
I20260928 02:15:58.415329 26947 fs_manager.cc:359] Metadata directory not provided
I20260928 02:15:58.415424 26947 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:15:58.415701 26947 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:15:58.418862 26947 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790561432251112-7085-0/minicluster-data/master-0/data/instance:
uuid: "4c2304e0f7534ec2ba1e48ca8525986f"
format_stamp: "Formatted at 2026-09-28 02:15:58 on dist-test-slave-86gv"
server_key: "6d636135dd50bd1e7609d6b555312df3"
server_key_iv: "9c89b8889283b7571057a11a2981dda1"
server_key_version: "encryptionkey@0"
I20260928 02:15:58.419688 26947 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790561432251112-7085-0/minicluster-data/master-0/wal/instance:
uuid: "4c2304e0f7534ec2ba1e48ca8525986f"
format_stamp: "Formatted at 2026-09-28 02:15:58 on dist-test-slave-86gv"
server_key: "6d636135dd50bd1e7609d6b555312df3"
server_key_iv: "9c89b8889283b7571057a11a2981dda1"
server_key_version: "encryptionkey@0"
I20260928 02:15:58.425271 26947 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.005s
I20260928 02:15:58.429141 26961 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:15:58.430742 26947 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260928 02:15:58.430948 26947 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790561432251112-7085-0/minicluster-data/master-0/data,/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790561432251112-7085-0/minicluster-data/master-0/wal
uuid: "4c2304e0f7534ec2ba1e48ca8525986f"
format_stamp: "Formatted at 2026-09-28 02:15:58 on dist-test-slave-86gv"
server_key: "6d636135dd50bd1e7609d6b555312df3"
server_key_iv: "9c89b8889283b7571057a11a2981dda1"
server_key_version: "encryptionkey@0"
I20260928 02:15:58.431200 26947 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790561432251112-7085-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790561432251112-7085-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790561432251112-7085-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:15:58.452499 26947 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:15:58.453545 26947 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:15:58.453886 26947 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:15:58.473441 26947 rpc_server.cc:307] RPC server started. Bound to: 127.6.235.126:32827
I20260928 02:15:58.473467 27012 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.235.126:32827 every 8 connection(s)
I20260928 02:15:58.474920 26947 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790561432251112-7085-0/minicluster-data/master-0/data/info.pb
I20260928 02:15:58.477716 7085 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTlaUuc/build/asan/bin/kudu as pid 26947
I20260928 02:15:58.477907 7085 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790561432251112-7085-0/minicluster-data/master-0/wal/instance
I20260928 02:15:58.478451 7085 external_mini_cluster.cc:1428] Setting key 47494b1ff77a97345c23fc9f7f1b07d9
I20260928 02:15:58.479534 27013 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:15:58.492872 27016 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:15:58.480611 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55516 (local address 127.6.235.126:32827)
0928 02:15:58.481675 (+ 1064us) server_negotiation.cc:206] Beginning negotiation
0928 02:15:58.481686 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 02:15:58.481735 (+ 49us) server_negotiation.cc:407] Connection header received
0928 02:15:58.483513 (+ 1778us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:15:58.483545 (+ 32us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:15:58.484064 (+ 519us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:15:58.484524 (+ 460us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:15:58.485146 (+ 622us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:15:58.486754 (+ 1608us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:15:58.488142 (+ 1388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:15:58.488879 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:15:58.489141 (+ 262us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:15:58.490386 (+ 1245us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:15:58.490393 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:15:58.490411 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 02:15:58.491034 (+ 623us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:15:58.491115 (+ 81us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 02:15:58.491138 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:15:58.491267 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0928 02:15:58.491502 (+ 235us) server_negotiation.cc:299] Negotiation successful
0928 02:15:58.491841 (+ 339us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":602,"thread_start_us":348,"threads_started":1}
I20260928 02:15:58.492972 27013 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4c2304e0f7534ec2ba1e48ca8525986f: Bootstrap starting.
I20260928 02:15:58.497077 27013 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4c2304e0f7534ec2ba1e48ca8525986f: Neither blocks nor log segments found. Creating new log.
I20260928 02:15:58.498183 27013 log.cc:824] T 00000000000000000000000000000000 P 4c2304e0f7534ec2ba1e48ca8525986f: Log is configured to *not* fsync() on all Append() calls
I20260928 02:15:58.501580 27013 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4c2304e0f7534ec2ba1e48ca8525986f: No bootstrap required, opened a new log
I20260928 02:15:58.509874 27013 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4c2304e0f7534ec2ba1e48ca8525986f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4c2304e0f7534ec2ba1e48ca8525986f" member_type: VOTER last_known_addr { host: "127.6.235.126" port: 32827 } }
I20260928 02:15:58.510319 27013 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4c2304e0f7534ec2ba1e48ca8525986f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:15:58.510452 27013 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4c2304e0f7534ec2ba1e48ca8525986f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4c2304e0f7534ec2ba1e48ca8525986f, State: Initialized, Role: FOLLOWER
I20260928 02:15:58.511092 27013 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4c2304e0f7534ec2ba1e48ca8525986f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4c2304e0f7534ec2ba1e48ca8525986f" member_type: VOTER last_known_addr { host: "127.6.235.126" port: 32827 } }
I20260928 02:15:58.511339 27013 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4c2304e0f7534ec2ba1e48ca8525986f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 02:15:58.511484 27013 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4c2304e0f7534ec2ba1e48ca8525986f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 02:15:58.511693 27013 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4c2304e0f7534ec2ba1e48ca8525986f [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:15:58.514138 27013 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4c2304e0f7534ec2ba1e48ca8525986f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4c2304e0f7534ec2ba1e48ca8525986f" member_type: VOTER last_known_addr { host: "127.6.235.126" port: 32827 } }
I20260928 02:15:58.514719 27013 leader_election.cc:302] T 00000000000000000000000000000000 P 4c2304e0f7534ec2ba1e48ca8525986f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4c2304e0f7534ec2ba1e48ca8525986f; no voters:
I20260928 02:15:58.515492 27013 leader_election.cc:288] T 00000000000000000000000000000000 P 4c2304e0f7534ec2ba1e48ca8525986f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 02:15:58.515722 27018 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4c2304e0f7534ec2ba1e48ca8525986f [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:15:58.516765 27018 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4c2304e0f7534ec2ba1e48ca8525986f [term 1 LEADER]: Becoming Leader. State: Replica: 4c2304e0f7534ec2ba1e48ca8525986f, State: Running, Role: LEADER
I20260928 02:15:58.517345 27018 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4c2304e0f7534ec2ba1e48ca8525986f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4c2304e0f7534ec2ba1e48ca8525986f" member_type: VOTER last_known_addr { host: "127.6.235.126" port: 32827 } }
I20260928 02:15:58.518065 27013 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4c2304e0f7534ec2ba1e48ca8525986f [sys.catalog]: configured and running, proceeding with master startup.
I20260928 02:15:58.520442 27019 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4c2304e0f7534ec2ba1e48ca8525986f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4c2304e0f7534ec2ba1e48ca8525986f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4c2304e0f7534ec2ba1e48ca8525986f" member_type: VOTER last_known_addr { host: "127.6.235.126" port: 32827 } } }
I20260928 02:15:58.520968 27019 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4c2304e0f7534ec2ba1e48ca8525986f [sys.catalog]: This master's current role is: LEADER
I20260928 02:15:58.520445 27020 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4c2304e0f7534ec2ba1e48ca8525986f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4c2304e0f7534ec2ba1e48ca8525986f. Latest consensus state: current_term: 1 leader_uuid: "4c2304e0f7534ec2ba1e48ca8525986f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4c2304e0f7534ec2ba1e48ca8525986f" member_type: VOTER last_known_addr { host: "127.6.235.126" port: 32827 } } }
I20260928 02:15:58.521486 27020 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4c2304e0f7534ec2ba1e48ca8525986f [sys.catalog]: This master's current role is: LEADER
I20260928 02:15:58.523816 27025 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 02:15:58.529448 27025 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 02:15:58.539243 27025 catalog_manager.cc:1347] Generated new cluster ID: ff1e617ebd0c4dc395c748cea6996684
I20260928 02:15:58.539372 27025 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 02:15:58.568568 27025 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 02:15:58.569941 27025 catalog_manager.cc:1504] Loading token signing keys...
I20260928 02:15:58.579180 27025 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4c2304e0f7534ec2ba1e48ca8525986f: Generated new TSK 0
I20260928 02:15:58.580360 27025 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 02:15:58.595444 7085 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTlaUuc/build/asan/bin/kudu
/tmp/dist-test-taskTlaUuc/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790561432251112-7085-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790561432251112-7085-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790561432251112-7085-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790561432251112-7085-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.235.65:0
--local_ip_for_outbound_sockets=127.6.235.65
--webserver_interface=127.6.235.65
--webserver_port=0
--tserver_master_addrs=127.6.235.126:32827
--builtin_ntp_servers=127.6.235.84:42579
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260928 02:15:58.960317 27037 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:15:58.960650 27037 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:15:58.960785 27037 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:15:58.968438 27037 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:15:58.968571 27037 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:15:58.968845 27037 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.235.65
I20260928 02:15:58.979714 27037 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.235.84:42579
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790561432251112-7085-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790561432251112-7085-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.235.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790561432251112-7085-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.6.235.65
--webserver_port=0
--tserver_master_addrs=127.6.235.126:32827
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.235.65
--log_dir=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790561432251112-7085-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 00:51:59 UTC on ccycloud.quasar-axgkwn.root.comops.site
build id 4676464
ASAN enabled
I20260928 02:15:58.981293 27037 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:15:58.983285 27037 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:15:58.997895 27042 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:15:58.998046 27043 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:15:58.998080 27045 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:15:58.998713 27037 server_base.cc:1034] running on GCE node
I20260928 02:15:58.999393 27037 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:15:59.000526 27037 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:15:59.001804 27037 hybrid_clock.cc:648] HybridClock initialized: now 1790561759001746 us; error 44 us; skew 500 ppm
I20260928 02:15:59.002230 27037 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260928 02:15:59.004750 27037 webserver.cc:460] Webserver started at http://127.6.235.65:43969/ using document root <none> and password file <none>
I20260928 02:15:59.005584 27037 fs_manager.cc:359] Metadata directory not provided
I20260928 02:15:59.005688 27037 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:15:59.006000 27037 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:15:59.009173 27037 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790561432251112-7085-0/minicluster-data/ts-0/data/instance:
uuid: "658c6c5db8e34fc8b0c1b608205e7542"
format_stamp: "Formatted at 2026-09-28 02:15:59 on dist-test-slave-86gv"
server_key: "6a2c0aaa45844af1eecfbf93c072c3d1"
server_key_iv: "2e90a4faf52531e8d4340eed56849d9e"
server_key_version: "encryptionkey@0"
I20260928 02:15:59.010079 27037 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790561432251112-7085-0/minicluster-data/ts-0/wal/instance:
uuid: "658c6c5db8e34fc8b0c1b608205e7542"
format_stamp: "Formatted at 2026-09-28 02:15:59 on dist-test-slave-86gv"
server_key: "6a2c0aaa45844af1eecfbf93c072c3d1"
server_key_iv: "2e90a4faf52531e8d4340eed56849d9e"
server_key_version: "encryptionkey@0"
I20260928 02:15:59.016147 27037 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.000s
I20260928 02:15:59.020448 27051 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:15:59.022085 27037 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260928 02:15:59.022289 27037 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790561432251112-7085-0/minicluster-data/ts-0/data,/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790561432251112-7085-0/minicluster-data/ts-0/wal
uuid: "658c6c5db8e34fc8b0c1b608205e7542"
format_stamp: "Formatted at 2026-09-28 02:15:59 on dist-test-slave-86gv"
server_key: "6a2c0aaa45844af1eecfbf93c072c3d1"
server_key_iv: "2e90a4faf52531e8d4340eed56849d9e"
server_key_version: "encryptionkey@0"
I20260928 02:15:59.022567 27037 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790561432251112-7085-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790561432251112-7085-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790561432251112-7085-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:15:59.045274 27037 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:15:59.046195 27037 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:15:59.046531 27037 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:15:59.047576 27037 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:15:59.049263 27037 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:15:59.049396 27037 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:15:59.049526 27037 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:15:59.049614 27037 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:15:59.085525 27037 rpc_server.cc:307] RPC server started. Bound to: 127.6.235.65:40115
I20260928 02:15:59.085635 27163 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.235.65:40115 every 8 connection(s)
I20260928 02:15:59.086962 27037 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790561432251112-7085-0/minicluster-data/ts-0/data/info.pb
I20260928 02:15:59.093719 7085 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTlaUuc/build/asan/bin/kudu as pid 27037
I20260928 02:15:59.093918 7085 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790561432251112-7085-0/minicluster-data/ts-0/wal/instance
I20260928 02:15:59.094409 7085 external_mini_cluster.cc:1428] Setting key 400620806fae60dbc4e595b9ea58e9fb
I20260928 02:15:59.097721 7085 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTlaUuc/build/asan/bin/kudu
/tmp/dist-test-taskTlaUuc/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790561432251112-7085-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790561432251112-7085-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790561432251112-7085-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790561432251112-7085-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.235.66:0
--local_ip_for_outbound_sockets=127.6.235.66
--webserver_interface=127.6.235.66
--webserver_port=0
--tserver_master_addrs=127.6.235.126:32827
--builtin_ntp_servers=127.6.235.84:42579
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260928 02:15:59.102751 27166 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:15:59.090023 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.235.65:45743 (local address 127.6.235.126:32827)
0928 02:15:59.090506 (+ 483us) server_negotiation.cc:206] Beginning negotiation
0928 02:15:59.090510 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:15:59.091630 (+ 1120us) server_negotiation.cc:407] Connection header received
0928 02:15:59.093603 (+ 1973us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:15:59.093610 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:15:59.093714 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:15:59.093840 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:15:59.095467 (+ 1627us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:15:59.096316 (+ 849us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:15:59.099125 (+ 2809us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:15:59.099636 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:15:59.099762 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:15:59.101837 (+ 2075us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:15:59.101843 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:15:59.101846 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 02:15:59.101869 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:15:59.101918 (+ 49us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 02:15:59.101933 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:15:59.102043 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0928 02:15:59.102324 (+ 281us) server_negotiation.cc:299] Negotiation successful
0928 02:15:59.102422 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":367,"thread_start_us":262,"threads_started":1}
I20260928 02:15:59.104344 27167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:15:59.090431 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.235.126:32827 (local address 127.6.235.65:45743)
0928 02:15:59.091446 (+ 1015us) negotiation.cc:107] Waiting for socket to connect
0928 02:15:59.091494 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0928 02:15:59.093290 (+ 1796us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:15:59.094147 (+ 857us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:15:59.094157 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:15:59.094261 (+ 104us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:15:59.095015 (+ 754us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:15:59.095043 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:15:59.096532 (+ 1489us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:15:59.096538 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:15:59.098857 (+ 2319us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:15:59.098881 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:15:59.099919 (+ 1038us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:15:59.099929 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:15:59.100314 (+ 385us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:15:59.101391 (+ 1077us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 02:15:59.101397 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:15:59.101458 (+ 61us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 02:15:59.101593 (+ 135us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:15:59.102059 (+ 466us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:15:59.102074 (+ 15us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:15:59.102086 (+ 12us) client_negotiation.cc:769] Sending connection context
0928 02:15:59.102283 (+ 197us) client_negotiation.cc:240] Negotiation successful
0928 02:15:59.102769 (+ 486us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":498,"thread_start_us":296,"threads_started":1}
I20260928 02:15:59.106487 27164 heartbeater.cc:347] Connected to a master server at 127.6.235.126:32827
I20260928 02:15:59.106874 27164 heartbeater.cc:464] Registering TS with master...
I20260928 02:15:59.107752 27164 heartbeater.cc:511] Master 127.6.235.126:32827 requested a full tablet report, sending...
I20260928 02:15:59.110056 26978 ts_manager.cc:194] Registered new tserver with Master: 658c6c5db8e34fc8b0c1b608205e7542 (127.6.235.65:40115)
I20260928 02:15:59.111594 26978 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.6.235.65:45743
W20260928 02:15:59.452174 27168 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:15:59.452574 27168 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:15:59.452750 27168 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:15:59.460561 27168 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:15:59.460738 27168 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:15:59.460976 27168 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.235.66
I20260928 02:15:59.470794 27168 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.235.84:42579
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790561432251112-7085-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790561432251112-7085-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.235.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790561432251112-7085-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.6.235.66
--webserver_port=0
--tserver_master_addrs=127.6.235.126:32827
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.235.66
--log_dir=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790561432251112-7085-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 00:51:59 UTC on ccycloud.quasar-axgkwn.root.comops.site
build id 4676464
ASAN enabled
I20260928 02:15:59.472200 27168 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:15:59.474701 27168 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:15:59.490564 27174 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:15:59.490621 27176 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:15:59.491583 27168 server_base.cc:1034] running on GCE node
W20260928 02:15:59.491757 27173 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:15:59.492597 27168 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:15:59.493870 27168 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:15:59.495139 27168 hybrid_clock.cc:648] HybridClock initialized: now 1790561759495094 us; error 42 us; skew 500 ppm
I20260928 02:15:59.495524 27168 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260928 02:15:59.498270 27168 webserver.cc:460] Webserver started at http://127.6.235.66:38409/ using document root <none> and password file <none>
I20260928 02:15:59.499060 27168 fs_manager.cc:359] Metadata directory not provided
I20260928 02:15:59.499188 27168 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:15:59.499547 27168 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:15:59.502723 27168 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790561432251112-7085-0/minicluster-data/ts-1/data/instance:
uuid: "dfcaab5cc5604949ad13b3cf0c28407e"
format_stamp: "Formatted at 2026-09-28 02:15:59 on dist-test-slave-86gv"
server_key: "a43d87d1acdba14c191d4c340ee0ef44"
server_key_iv: "15f3294666e791e523cd8072150e1fad"
server_key_version: "encryptionkey@0"
I20260928 02:15:59.503573 27168 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790561432251112-7085-0/minicluster-data/ts-1/wal/instance:
uuid: "dfcaab5cc5604949ad13b3cf0c28407e"
format_stamp: "Formatted at 2026-09-28 02:15:59 on dist-test-slave-86gv"
server_key: "a43d87d1acdba14c191d4c340ee0ef44"
server_key_iv: "15f3294666e791e523cd8072150e1fad"
server_key_version: "encryptionkey@0"
I20260928 02:15:59.509173 27168 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.004s
I20260928 02:15:59.512882 27182 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:15:59.514387 27168 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 02:15:59.514570 27168 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790561432251112-7085-0/minicluster-data/ts-1/data,/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790561432251112-7085-0/minicluster-data/ts-1/wal
uuid: "dfcaab5cc5604949ad13b3cf0c28407e"
format_stamp: "Formatted at 2026-09-28 02:15:59 on dist-test-slave-86gv"
server_key: "a43d87d1acdba14c191d4c340ee0ef44"
server_key_iv: "15f3294666e791e523cd8072150e1fad"
server_key_version: "encryptionkey@0"
I20260928 02:15:59.514818 27168 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790561432251112-7085-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790561432251112-7085-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790561432251112-7085-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:15:59.528831 27168 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:15:59.529839 27168 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:15:59.530128 27168 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:15:59.531291 27168 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:15:59.532759 27168 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:15:59.532847 27168 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:15:59.533007 27168 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:15:59.533092 27168 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:15:59.568938 27168 rpc_server.cc:307] RPC server started. Bound to: 127.6.235.66:33213
I20260928 02:15:59.569017 27294 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.235.66:33213 every 8 connection(s)
I20260928 02:15:59.570360 27168 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790561432251112-7085-0/minicluster-data/ts-1/data/info.pb
I20260928 02:15:59.580206 7085 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTlaUuc/build/asan/bin/kudu as pid 27168
I20260928 02:15:59.580382 7085 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790561432251112-7085-0/minicluster-data/ts-1/wal/instance
I20260928 02:15:59.580955 7085 external_mini_cluster.cc:1428] Setting key 8e17adfb86f18b663337661e24cac56e
I20260928 02:15:59.584249 27166 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:15:59.573046 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.235.66:34141 (local address 127.6.235.126:32827)
0928 02:15:59.573269 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0928 02:15:59.573276 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 02:15:59.574843 (+ 1567us) server_negotiation.cc:407] Connection header received
0928 02:15:59.576626 (+ 1783us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:15:59.576631 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:15:59.576701 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:15:59.576793 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:15:59.578101 (+ 1308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:15:59.578835 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:15:59.581019 (+ 2184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:15:59.581482 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:15:59.581644 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:15:59.583442 (+ 1798us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:15:59.583448 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:15:59.583451 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 02:15:59.583470 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:15:59.583505 (+ 35us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 02:15:59.583518 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:15:59.583585 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0928 02:15:59.583908 (+ 323us) server_negotiation.cc:299] Negotiation successful
0928 02:15:59.584022 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260928 02:15:59.584506 7085 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTlaUuc/build/asan/bin/kudu
/tmp/dist-test-taskTlaUuc/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790561432251112-7085-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790561432251112-7085-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790561432251112-7085-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790561432251112-7085-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.235.67:0
--local_ip_for_outbound_sockets=127.6.235.67
--webserver_interface=127.6.235.67
--webserver_port=0
--tserver_master_addrs=127.6.235.126:32827
--builtin_ntp_servers=127.6.235.84:42579
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260928 02:15:59.586055 27297 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:15:59.573698 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.235.126:32827 (local address 127.6.235.66:34141)
0928 02:15:59.574677 (+ 979us) negotiation.cc:107] Waiting for socket to connect
0928 02:15:59.574721 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0928 02:15:59.576347 (+ 1626us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:15:59.577178 (+ 831us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:15:59.577187 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:15:59.577289 (+ 102us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:15:59.577925 (+ 636us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:15:59.577943 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:15:59.578995 (+ 1052us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:15:59.579000 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:15:59.580772 (+ 1772us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:15:59.580789 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:15:59.581760 (+ 971us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:15:59.581769 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:15:59.582091 (+ 322us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:15:59.583000 (+ 909us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 02:15:59.583005 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:15:59.583079 (+ 74us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 02:15:59.583199 (+ 120us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:15:59.583624 (+ 425us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:15:59.583634 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:15:59.583643 (+ 9us) client_negotiation.cc:769] Sending connection context
0928 02:15:59.583870 (+ 227us) client_negotiation.cc:240] Negotiation successful
0928 02:15:59.584372 (+ 502us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":503,"thread_start_us":322,"threads_started":1}
I20260928 02:15:59.587743 27295 heartbeater.cc:347] Connected to a master server at 127.6.235.126:32827
I20260928 02:15:59.588095 27295 heartbeater.cc:464] Registering TS with master...
I20260928 02:15:59.589143 27295 heartbeater.cc:511] Master 127.6.235.126:32827 requested a full tablet report, sending...
I20260928 02:15:59.591105 26978 ts_manager.cc:194] Registered new tserver with Master: dfcaab5cc5604949ad13b3cf0c28407e (127.6.235.66:33213)
I20260928 02:15:59.592034 26978 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.6.235.66:34141
W20260928 02:15:59.940680 27298 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:15:59.941068 27298 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:15:59.941232 27298 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:15:59.948848 27298 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:15:59.949076 27298 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:15:59.949246 27298 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.235.67
I20260928 02:15:59.958900 27298 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.235.84:42579
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790561432251112-7085-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790561432251112-7085-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.235.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790561432251112-7085-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.6.235.67
--webserver_port=0
--tserver_master_addrs=127.6.235.126:32827
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.235.67
--log_dir=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790561432251112-7085-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 00:51:59 UTC on ccycloud.quasar-axgkwn.root.comops.site
build id 4676464
ASAN enabled
I20260928 02:15:59.960767 27298 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:15:59.962858 27298 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:15:59.977200 27303 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:15:59.977106 27304 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:15:59.977177 27306 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:15:59.982797 27298 server_base.cc:1034] running on GCE node
I20260928 02:15:59.983511 27298 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:15:59.984576 27298 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:15:59.985836 27298 hybrid_clock.cc:648] HybridClock initialized: now 1790561759985731 us; error 99 us; skew 500 ppm
I20260928 02:15:59.986409 27298 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260928 02:15:59.988845 27298 webserver.cc:460] Webserver started at http://127.6.235.67:41091/ using document root <none> and password file <none>
I20260928 02:15:59.989745 27298 fs_manager.cc:359] Metadata directory not provided
I20260928 02:15:59.990101 27298 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:15:59.990566 27298 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:15:59.994182 27298 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790561432251112-7085-0/minicluster-data/ts-2/data/instance:
uuid: "1653d1fcbf534f33a82d04943fa7ed82"
format_stamp: "Formatted at 2026-09-28 02:15:59 on dist-test-slave-86gv"
server_key: "c42c87795964a97fc6845c82228241b1"
server_key_iv: "1fe72c007c45cb02a59bb962d5b47211"
server_key_version: "encryptionkey@0"
I20260928 02:15:59.995096 27298 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790561432251112-7085-0/minicluster-data/ts-2/wal/instance:
uuid: "1653d1fcbf534f33a82d04943fa7ed82"
format_stamp: "Formatted at 2026-09-28 02:15:59 on dist-test-slave-86gv"
server_key: "c42c87795964a97fc6845c82228241b1"
server_key_iv: "1fe72c007c45cb02a59bb962d5b47211"
server_key_version: "encryptionkey@0"
I20260928 02:16:00.001165 27298 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.001s
I20260928 02:16:00.005313 27312 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:16:00.006809 27298 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260928 02:16:00.007011 27298 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790561432251112-7085-0/minicluster-data/ts-2/data,/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790561432251112-7085-0/minicluster-data/ts-2/wal
uuid: "1653d1fcbf534f33a82d04943fa7ed82"
format_stamp: "Formatted at 2026-09-28 02:15:59 on dist-test-slave-86gv"
server_key: "c42c87795964a97fc6845c82228241b1"
server_key_iv: "1fe72c007c45cb02a59bb962d5b47211"
server_key_version: "encryptionkey@0"
I20260928 02:16:00.007253 27298 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790561432251112-7085-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790561432251112-7085-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790561432251112-7085-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:16:00.034060 27298 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:16:00.035000 27298 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:16:00.035326 27298 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:16:00.036378 27298 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:16:00.038053 27298 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:16:00.038179 27298 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:16:00.038321 27298 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:16:00.038414 27298 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:16:00.074077 27298 rpc_server.cc:307] RPC server started. Bound to: 127.6.235.67:44055
I20260928 02:16:00.074163 27424 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.235.67:44055 every 8 connection(s)
I20260928 02:16:00.075484 27298 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790561432251112-7085-0/minicluster-data/ts-2/data/info.pb
I20260928 02:16:00.084419 7085 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTlaUuc/build/asan/bin/kudu as pid 27298
I20260928 02:16:00.084589 7085 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790561432251112-7085-0/minicluster-data/ts-2/wal/instance
I20260928 02:16:00.085207 7085 external_mini_cluster.cc:1428] Setting key ee06ad53734e8355ecae76a808a86b9b
I20260928 02:16:00.089795 27166 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:16:00.078205 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.235.67:41537 (local address 127.6.235.126:32827)
0928 02:16:00.078385 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0928 02:16:00.078390 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 02:16:00.079930 (+ 1540us) server_negotiation.cc:407] Connection header received
0928 02:16:00.081891 (+ 1961us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:16:00.081896 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:16:00.081969 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:16:00.082100 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:16:00.083212 (+ 1112us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:16:00.084053 (+ 841us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:16:00.086544 (+ 2491us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:16:00.087199 (+ 655us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:16:00.087323 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:16:00.088881 (+ 1558us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:16:00.088888 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:16:00.088920 (+ 32us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 02:16:00.088944 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:16:00.088987 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 02:16:00.089005 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:16:00.089101 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0928 02:16:00.089303 (+ 202us) server_negotiation.cc:299] Negotiation successful
0928 02:16:00.089415 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260928 02:16:00.090952 27427 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:16:00.078727 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.235.126:32827 (local address 127.6.235.67:41537)
0928 02:16:00.079777 (+ 1050us) negotiation.cc:107] Waiting for socket to connect
0928 02:16:00.079815 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0928 02:16:00.081617 (+ 1802us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:16:00.082310 (+ 693us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:16:00.082319 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:16:00.082416 (+ 97us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:16:00.083032 (+ 616us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:16:00.083050 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:16:00.084226 (+ 1176us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:16:00.084240 (+ 14us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:16:00.086298 (+ 2058us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:16:00.086315 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:16:00.087439 (+ 1124us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:16:00.087446 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:16:00.087776 (+ 330us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:16:00.088564 (+ 788us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 02:16:00.088567 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:16:00.088606 (+ 39us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 02:16:00.088690 (+ 84us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:16:00.089096 (+ 406us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:16:00.089104 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:16:00.089110 (+ 6us) client_negotiation.cc:769] Sending connection context
0928 02:16:00.089257 (+ 147us) client_negotiation.cc:240] Negotiation successful
0928 02:16:00.089654 (+ 397us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":576,"thread_start_us":384,"threads_started":1}
I20260928 02:16:00.092429 27425 heartbeater.cc:347] Connected to a master server at 127.6.235.126:32827
I20260928 02:16:00.092741 27425 heartbeater.cc:464] Registering TS with master...
I20260928 02:16:00.093604 27425 heartbeater.cc:511] Master 127.6.235.126:32827 requested a full tablet report, sending...
I20260928 02:16:00.094961 26978 ts_manager.cc:194] Registered new tserver with Master: 1653d1fcbf534f33a82d04943fa7ed82 (127.6.235.67:44055)
I20260928 02:16:00.095764 26978 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.6.235.67:41537
I20260928 02:16:00.100873 7085 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 02:16:00.111809 27166 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:16:00.105622 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55522 (local address 127.6.235.126:32827)
0928 02:16:00.105846 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0928 02:16:00.105850 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:16:00.106314 (+ 464us) server_negotiation.cc:407] Connection header received
0928 02:16:00.106456 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:16:00.106460 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:16:00.106530 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:16:00.106614 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:16:00.107114 (+ 500us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:16:00.107814 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:16:00.109343 (+ 1529us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:16:00.109768 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:16:00.109869 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:16:00.111104 (+ 1235us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:16:00.111109 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:16:00.111112 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 02:16:00.111130 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:16:00.111163 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 02:16:00.111175 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:16:00.111246 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0928 02:16:00.111413 (+ 167us) server_negotiation.cc:299] Negotiation successful
0928 02:16:00.111540 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260928 02:16:00.115223 27164 heartbeater.cc:503] Master 127.6.235.126:32827 was elected leader, sending a full tablet report...
I20260928 02:16:00.116461 26978 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55522:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 02:16:00.123761 26978 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 02:16:00.149991 27441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:16:00.138492 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42516 (local address 127.6.235.66:33213)
0928 02:16:00.139296 (+ 804us) server_negotiation.cc:206] Beginning negotiation
0928 02:16:00.139309 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0928 02:16:00.141535 (+ 2226us) server_negotiation.cc:407] Connection header received
0928 02:16:00.141885 (+ 350us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:16:00.141897 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:16:00.142112 (+ 215us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:16:00.142336 (+ 224us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:16:00.143420 (+ 1084us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:16:00.144454 (+ 1034us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:16:00.147444 (+ 2990us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:16:00.148689 (+ 1245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:16:00.148771 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:16:00.149037 (+ 266us) server_negotiation.cc:1035] Waiting for connection context
0928 02:16:00.149483 (+ 446us) server_negotiation.cc:299] Negotiation successful
0928 02:16:00.149656 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":589,"thread_start_us":465,"threads_started":1}
I20260928 02:16:00.150058 27440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:16:00.137917 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.235.66:33213 (local address 127.0.0.1:42516)
0928 02:16:00.141354 (+ 3437us) negotiation.cc:107] Waiting for socket to connect
0928 02:16:00.141394 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0928 02:16:00.141680 (+ 286us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:16:00.142653 (+ 973us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:16:00.142657 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:16:00.142687 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:16:00.143196 (+ 509us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:16:00.143214 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:16:00.144603 (+ 1389us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:16:00.144606 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:16:00.147214 (+ 2608us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:16:00.147229 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:16:00.149087 (+ 1858us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:16:00.149094 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:16:00.149249 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:16:00.149282 (+ 33us) client_negotiation.cc:769] Sending connection context
0928 02:16:00.149435 (+ 153us) client_negotiation.cc:240] Negotiation successful
0928 02:16:00.149624 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3259,"mutex_wait_us":28,"thread_start_us":3479,"threads_started":1}
I20260928 02:16:00.150612 27437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:16:00.137078 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.235.65:40115 (local address 127.0.0.1:50504)
0928 02:16:00.138170 (+ 1092us) negotiation.cc:107] Waiting for socket to connect
0928 02:16:00.138233 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0928 02:16:00.138505 (+ 272us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:16:00.142550 (+ 4045us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:16:00.142559 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:16:00.142595 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:16:00.143131 (+ 536us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:16:00.143152 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:16:00.145463 (+ 2311us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:16:00.145468 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:16:00.147841 (+ 2373us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:16:00.147853 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:16:00.149811 (+ 1958us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:16:00.149817 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:16:00.149998 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:16:00.150035 (+ 37us) client_negotiation.cc:769] Sending connection context
0928 02:16:00.150104 (+ 69us) client_negotiation.cc:240] Negotiation successful
0928 02:16:00.150224 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":697,"mutex_wait_us":232,"thread_start_us":496,"threads_started":1}
I20260928 02:16:00.151953 27438 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:16:00.137492 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50504 (local address 127.6.235.65:40115)
0928 02:16:00.141967 (+ 4475us) server_negotiation.cc:206] Beginning negotiation
0928 02:16:00.141987 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0928 02:16:00.142011 (+ 24us) server_negotiation.cc:407] Connection header received
0928 02:16:00.142099 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:16:00.142105 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:16:00.142314 (+ 209us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:16:00.143415 (+ 1101us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:16:00.143672 (+ 257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:16:00.145317 (+ 1645us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:16:00.148053 (+ 2736us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:16:00.149591 (+ 1538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:16:00.151044 (+ 1453us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:16:00.151308 (+ 264us) server_negotiation.cc:1035] Waiting for connection context
0928 02:16:00.151384 (+ 76us) server_negotiation.cc:299] Negotiation successful
0928 02:16:00.151581 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4300,"thread_start_us":4166,"threads_started":1}
I20260928 02:16:00.152207 27439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:16:00.138423 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.235.67:44055 (local address 127.0.0.1:55808)
0928 02:16:00.141709 (+ 3286us) negotiation.cc:107] Waiting for socket to connect
0928 02:16:00.141729 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0928 02:16:00.141855 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:16:00.142802 (+ 947us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:16:00.142806 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:16:00.142829 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:16:00.143308 (+ 479us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:16:00.143322 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:16:00.145347 (+ 2025us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:16:00.145352 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:16:00.148029 (+ 2677us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:16:00.148043 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:16:00.151445 (+ 3402us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:16:00.151455 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:16:00.151617 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:16:00.151646 (+ 29us) client_negotiation.cc:769] Sending connection context
0928 02:16:00.151745 (+ 99us) client_negotiation.cc:240] Negotiation successful
0928 02:16:00.151870 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3143,"mutex_wait_us":58,"thread_start_us":2595,"threads_started":1}
I20260928 02:16:00.152530 27442 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:16:00.138952 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55808 (local address 127.6.235.67:44055)
0928 02:16:00.140803 (+ 1851us) server_negotiation.cc:206] Beginning negotiation
0928 02:16:00.140816 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0928 02:16:00.141979 (+ 1163us) server_negotiation.cc:407] Connection header received
0928 02:16:00.142084 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:16:00.142091 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:16:00.142314 (+ 223us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:16:00.142518 (+ 204us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:16:00.143530 (+ 1012us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:16:00.145199 (+ 1669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:16:00.148216 (+ 3017us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:16:00.149591 (+ 1375us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:16:00.149743 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:16:00.150027 (+ 284us) server_negotiation.cc:1035] Waiting for connection context
0928 02:16:00.151825 (+ 1798us) server_negotiation.cc:299] Negotiation successful
0928 02:16:00.151996 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1630,"thread_start_us":1442,"threads_started":1}
I20260928 02:16:00.154615 27230 tablet_service.cc:1467] Processing CreateTablet for tablet f71af20203ad4a9fbc9459c6bb1aec7a (DEFAULT_TABLE table=test-table [id=4d9e2c5af4304c2d840769a5c1e65d03]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:16:00.156240 27230 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f71af20203ad4a9fbc9459c6bb1aec7a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:16:00.156104 27099 tablet_service.cc:1467] Processing CreateTablet for tablet f71af20203ad4a9fbc9459c6bb1aec7a (DEFAULT_TABLE table=test-table [id=4d9e2c5af4304c2d840769a5c1e65d03]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:16:00.156462 27360 tablet_service.cc:1467] Processing CreateTablet for tablet f71af20203ad4a9fbc9459c6bb1aec7a (DEFAULT_TABLE table=test-table [id=4d9e2c5af4304c2d840769a5c1e65d03]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:16:00.157618 27099 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f71af20203ad4a9fbc9459c6bb1aec7a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:16:00.157903 27360 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f71af20203ad4a9fbc9459c6bb1aec7a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:16:00.169610 27443 tablet_bootstrap.cc:492] T f71af20203ad4a9fbc9459c6bb1aec7a P 658c6c5db8e34fc8b0c1b608205e7542: Bootstrap starting.
I20260928 02:16:00.174644 27443 tablet_bootstrap.cc:654] T f71af20203ad4a9fbc9459c6bb1aec7a P 658c6c5db8e34fc8b0c1b608205e7542: Neither blocks nor log segments found. Creating new log.
I20260928 02:16:00.175751 27443 log.cc:824] T f71af20203ad4a9fbc9459c6bb1aec7a P 658c6c5db8e34fc8b0c1b608205e7542: Log is configured to *not* fsync() on all Append() calls
I20260928 02:16:00.179155 27443 tablet_bootstrap.cc:492] T f71af20203ad4a9fbc9459c6bb1aec7a P 658c6c5db8e34fc8b0c1b608205e7542: No bootstrap required, opened a new log
I20260928 02:16:00.179472 27443 ts_tablet_manager.cc:1397] T f71af20203ad4a9fbc9459c6bb1aec7a P 658c6c5db8e34fc8b0c1b608205e7542: Time spent bootstrapping tablet: real 0.010s user 0.006s sys 0.000s
I20260928 02:16:00.181525 27444 tablet_bootstrap.cc:492] T f71af20203ad4a9fbc9459c6bb1aec7a P dfcaab5cc5604949ad13b3cf0c28407e: Bootstrap starting.
I20260928 02:16:00.184576 27444 tablet_bootstrap.cc:654] T f71af20203ad4a9fbc9459c6bb1aec7a P dfcaab5cc5604949ad13b3cf0c28407e: Neither blocks nor log segments found. Creating new log.
I20260928 02:16:00.185161 27445 tablet_bootstrap.cc:492] T f71af20203ad4a9fbc9459c6bb1aec7a P 1653d1fcbf534f33a82d04943fa7ed82: Bootstrap starting.
I20260928 02:16:00.185813 27444 log.cc:824] T f71af20203ad4a9fbc9459c6bb1aec7a P dfcaab5cc5604949ad13b3cf0c28407e: Log is configured to *not* fsync() on all Append() calls
I20260928 02:16:00.188619 27444 tablet_bootstrap.cc:492] T f71af20203ad4a9fbc9459c6bb1aec7a P dfcaab5cc5604949ad13b3cf0c28407e: No bootstrap required, opened a new log
I20260928 02:16:00.188339 27443 raft_consensus.cc:348] T f71af20203ad4a9fbc9459c6bb1aec7a P 658c6c5db8e34fc8b0c1b608205e7542 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "658c6c5db8e34fc8b0c1b608205e7542" member_type: VOTER last_known_addr { host: "127.6.235.65" port: 40115 } } peers { permanent_uuid: "1653d1fcbf534f33a82d04943fa7ed82" member_type: VOTER last_known_addr { host: "127.6.235.67" port: 44055 } } peers { permanent_uuid: "dfcaab5cc5604949ad13b3cf0c28407e" member_type: VOTER last_known_addr { host: "127.6.235.66" port: 33213 } }
I20260928 02:16:00.188822 27443 raft_consensus.cc:374] T f71af20203ad4a9fbc9459c6bb1aec7a P 658c6c5db8e34fc8b0c1b608205e7542 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:16:00.188983 27444 ts_tablet_manager.cc:1397] T f71af20203ad4a9fbc9459c6bb1aec7a P dfcaab5cc5604949ad13b3cf0c28407e: Time spent bootstrapping tablet: real 0.008s user 0.000s sys 0.006s
I20260928 02:16:00.189004 27445 tablet_bootstrap.cc:654] T f71af20203ad4a9fbc9459c6bb1aec7a P 1653d1fcbf534f33a82d04943fa7ed82: Neither blocks nor log segments found. Creating new log.
I20260928 02:16:00.189004 27443 raft_consensus.cc:729] T f71af20203ad4a9fbc9459c6bb1aec7a P 658c6c5db8e34fc8b0c1b608205e7542 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 658c6c5db8e34fc8b0c1b608205e7542, State: Initialized, Role: FOLLOWER
I20260928 02:16:00.189716 27443 consensus_queue.cc:260] T f71af20203ad4a9fbc9459c6bb1aec7a P 658c6c5db8e34fc8b0c1b608205e7542 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "658c6c5db8e34fc8b0c1b608205e7542" member_type: VOTER last_known_addr { host: "127.6.235.65" port: 40115 } } peers { permanent_uuid: "1653d1fcbf534f33a82d04943fa7ed82" member_type: VOTER last_known_addr { host: "127.6.235.67" port: 44055 } } peers { permanent_uuid: "dfcaab5cc5604949ad13b3cf0c28407e" member_type: VOTER last_known_addr { host: "127.6.235.66" port: 33213 } }
I20260928 02:16:00.190282 27445 log.cc:824] T f71af20203ad4a9fbc9459c6bb1aec7a P 1653d1fcbf534f33a82d04943fa7ed82: Log is configured to *not* fsync() on all Append() calls
I20260928 02:16:00.191752 27443 ts_tablet_manager.cc:1428] T f71af20203ad4a9fbc9459c6bb1aec7a P 658c6c5db8e34fc8b0c1b608205e7542: Time spent starting tablet: real 0.012s user 0.013s sys 0.000s
I20260928 02:16:00.195520 27445 tablet_bootstrap.cc:492] T f71af20203ad4a9fbc9459c6bb1aec7a P 1653d1fcbf534f33a82d04943fa7ed82: No bootstrap required, opened a new log
I20260928 02:16:00.195829 27445 ts_tablet_manager.cc:1397] T f71af20203ad4a9fbc9459c6bb1aec7a P 1653d1fcbf534f33a82d04943fa7ed82: Time spent bootstrapping tablet: real 0.011s user 0.005s sys 0.003s
I20260928 02:16:00.198830 27444 raft_consensus.cc:348] T f71af20203ad4a9fbc9459c6bb1aec7a P dfcaab5cc5604949ad13b3cf0c28407e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "658c6c5db8e34fc8b0c1b608205e7542" member_type: VOTER last_known_addr { host: "127.6.235.65" port: 40115 } } peers { permanent_uuid: "1653d1fcbf534f33a82d04943fa7ed82" member_type: VOTER last_known_addr { host: "127.6.235.67" port: 44055 } } peers { permanent_uuid: "dfcaab5cc5604949ad13b3cf0c28407e" member_type: VOTER last_known_addr { host: "127.6.235.66" port: 33213 } }
I20260928 02:16:00.199260 27444 raft_consensus.cc:374] T f71af20203ad4a9fbc9459c6bb1aec7a P dfcaab5cc5604949ad13b3cf0c28407e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:16:00.199370 27444 raft_consensus.cc:729] T f71af20203ad4a9fbc9459c6bb1aec7a P dfcaab5cc5604949ad13b3cf0c28407e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dfcaab5cc5604949ad13b3cf0c28407e, State: Initialized, Role: FOLLOWER
I20260928 02:16:00.200043 27444 consensus_queue.cc:260] T f71af20203ad4a9fbc9459c6bb1aec7a P dfcaab5cc5604949ad13b3cf0c28407e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "658c6c5db8e34fc8b0c1b608205e7542" member_type: VOTER last_known_addr { host: "127.6.235.65" port: 40115 } } peers { permanent_uuid: "1653d1fcbf534f33a82d04943fa7ed82" member_type: VOTER last_known_addr { host: "127.6.235.67" port: 44055 } } peers { permanent_uuid: "dfcaab5cc5604949ad13b3cf0c28407e" member_type: VOTER last_known_addr { host: "127.6.235.66" port: 33213 } }
I20260928 02:16:00.201249 27295 heartbeater.cc:503] Master 127.6.235.126:32827 was elected leader, sending a full tablet report...
I20260928 02:16:00.201900 27444 ts_tablet_manager.cc:1428] T f71af20203ad4a9fbc9459c6bb1aec7a P dfcaab5cc5604949ad13b3cf0c28407e: Time spent starting tablet: real 0.013s user 0.007s sys 0.005s
I20260928 02:16:00.204113 27445 raft_consensus.cc:348] T f71af20203ad4a9fbc9459c6bb1aec7a P 1653d1fcbf534f33a82d04943fa7ed82 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "658c6c5db8e34fc8b0c1b608205e7542" member_type: VOTER last_known_addr { host: "127.6.235.65" port: 40115 } } peers { permanent_uuid: "1653d1fcbf534f33a82d04943fa7ed82" member_type: VOTER last_known_addr { host: "127.6.235.67" port: 44055 } } peers { permanent_uuid: "dfcaab5cc5604949ad13b3cf0c28407e" member_type: VOTER last_known_addr { host: "127.6.235.66" port: 33213 } }
I20260928 02:16:00.204586 27445 raft_consensus.cc:374] T f71af20203ad4a9fbc9459c6bb1aec7a P 1653d1fcbf534f33a82d04943fa7ed82 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:16:00.204706 27445 raft_consensus.cc:729] T f71af20203ad4a9fbc9459c6bb1aec7a P 1653d1fcbf534f33a82d04943fa7ed82 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1653d1fcbf534f33a82d04943fa7ed82, State: Initialized, Role: FOLLOWER
I20260928 02:16:00.205761 27445 consensus_queue.cc:260] T f71af20203ad4a9fbc9459c6bb1aec7a P 1653d1fcbf534f33a82d04943fa7ed82 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "658c6c5db8e34fc8b0c1b608205e7542" member_type: VOTER last_known_addr { host: "127.6.235.65" port: 40115 } } peers { permanent_uuid: "1653d1fcbf534f33a82d04943fa7ed82" member_type: VOTER last_known_addr { host: "127.6.235.67" port: 44055 } } peers { permanent_uuid: "dfcaab5cc5604949ad13b3cf0c28407e" member_type: VOTER last_known_addr { host: "127.6.235.66" port: 33213 } }
I20260928 02:16:00.206983 27425 heartbeater.cc:503] Master 127.6.235.126:32827 was elected leader, sending a full tablet report...
I20260928 02:16:00.207126 27448 raft_consensus.cc:482] T f71af20203ad4a9fbc9459c6bb1aec7a P 658c6c5db8e34fc8b0c1b608205e7542 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 02:16:00.207409 27448 raft_consensus.cc:504] T f71af20203ad4a9fbc9459c6bb1aec7a P 658c6c5db8e34fc8b0c1b608205e7542 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "658c6c5db8e34fc8b0c1b608205e7542" member_type: VOTER last_known_addr { host: "127.6.235.65" port: 40115 } } peers { permanent_uuid: "1653d1fcbf534f33a82d04943fa7ed82" member_type: VOTER last_known_addr { host: "127.6.235.67" port: 44055 } } peers { permanent_uuid: "dfcaab5cc5604949ad13b3cf0c28407e" member_type: VOTER last_known_addr { host: "127.6.235.66" port: 33213 } }
I20260928 02:16:00.207793 27445 ts_tablet_manager.cc:1428] T f71af20203ad4a9fbc9459c6bb1aec7a P 1653d1fcbf534f33a82d04943fa7ed82: Time spent starting tablet: real 0.012s user 0.012s sys 0.001s
I20260928 02:16:00.209609 27448 leader_election.cc:288] T f71af20203ad4a9fbc9459c6bb1aec7a P 658c6c5db8e34fc8b0c1b608205e7542 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1653d1fcbf534f33a82d04943fa7ed82 (127.6.235.67:44055), dfcaab5cc5604949ad13b3cf0c28407e (127.6.235.66:33213)
I20260928 02:16:00.217087 27442 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:16:00.209821 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.235.65:41887 (local address 127.6.235.67:44055)
0928 02:16:00.210066 (+ 245us) server_negotiation.cc:206] Beginning negotiation
0928 02:16:00.210072 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 02:16:00.211113 (+ 1041us) server_negotiation.cc:407] Connection header received
0928 02:16:00.211382 (+ 269us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:16:00.211390 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:16:00.211477 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:16:00.211568 (+ 91us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:16:00.212321 (+ 753us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:16:00.213384 (+ 1063us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:16:00.215342 (+ 1958us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:16:00.216120 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:16:00.216302 (+ 182us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:16:00.216376 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0928 02:16:00.216664 (+ 288us) server_negotiation.cc:299] Negotiation successful
0928 02:16:00.216781 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":104}
I20260928 02:16:00.217181 27453 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:16:00.209749 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.235.67:44055 (local address 127.6.235.65:41887)
0928 02:16:00.210979 (+ 1230us) negotiation.cc:107] Waiting for socket to connect
0928 02:16:00.210999 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0928 02:16:00.211187 (+ 188us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:16:00.211742 (+ 555us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:16:00.211747 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:16:00.211772 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:16:00.212172 (+ 400us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:16:00.212190 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:16:00.213507 (+ 1317us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:16:00.213514 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:16:00.215124 (+ 1610us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:16:00.215137 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:16:00.216305 (+ 1168us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:16:00.216311 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:16:00.216492 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:16:00.216516 (+ 24us) client_negotiation.cc:769] Sending connection context
0928 02:16:00.216626 (+ 110us) client_negotiation.cc:240] Negotiation successful
0928 02:16:00.216764 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1069,"mutex_wait_us":68,"thread_start_us":1790,"threads_started":1}
I20260928 02:16:00.217231 27441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:16:00.210219 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.235.65:57153 (local address 127.6.235.66:33213)
0928 02:16:00.210418 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0928 02:16:00.210424 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 02:16:00.211920 (+ 1496us) server_negotiation.cc:407] Connection header received
0928 02:16:00.212214 (+ 294us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:16:00.212220 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:16:00.212330 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:16:00.212428 (+ 98us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:16:00.213031 (+ 603us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:16:00.213701 (+ 670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:16:00.215491 (+ 1790us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:16:00.216239 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:16:00.216403 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:16:00.216476 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0928 02:16:00.216823 (+ 347us) server_negotiation.cc:299] Negotiation successful
0928 02:16:00.216990 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260928 02:16:00.217270 27452 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:16:00.210216 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.235.66:33213 (local address 127.6.235.65:57153)
0928 02:16:00.211806 (+ 1590us) negotiation.cc:107] Waiting for socket to connect
0928 02:16:00.211835 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0928 02:16:00.212020 (+ 185us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:16:00.212466 (+ 446us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:16:00.212472 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:16:00.212499 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:16:00.212842 (+ 343us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:16:00.212854 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:16:00.213846 (+ 992us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:16:00.213851 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:16:00.215312 (+ 1461us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:16:00.215324 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:16:00.216439 (+ 1115us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:16:00.216445 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:16:00.216633 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:16:00.216679 (+ 46us) client_negotiation.cc:769] Sending connection context
0928 02:16:00.216788 (+ 109us) client_negotiation.cc:240] Negotiation successful
0928 02:16:00.216933 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1468,"thread_start_us":401,"threads_started":1}
I20260928 02:16:00.218223 27250 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f71af20203ad4a9fbc9459c6bb1aec7a" candidate_uuid: "658c6c5db8e34fc8b0c1b608205e7542" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dfcaab5cc5604949ad13b3cf0c28407e" is_pre_election: true
I20260928 02:16:00.218434 27380 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f71af20203ad4a9fbc9459c6bb1aec7a" candidate_uuid: "658c6c5db8e34fc8b0c1b608205e7542" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1653d1fcbf534f33a82d04943fa7ed82" is_pre_election: true
I20260928 02:16:00.218884 27250 raft_consensus.cc:2454] T f71af20203ad4a9fbc9459c6bb1aec7a P dfcaab5cc5604949ad13b3cf0c28407e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 658c6c5db8e34fc8b0c1b608205e7542 in term 0.
I20260928 02:16:00.219012 27380 raft_consensus.cc:2454] T f71af20203ad4a9fbc9459c6bb1aec7a P 1653d1fcbf534f33a82d04943fa7ed82 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 658c6c5db8e34fc8b0c1b608205e7542 in term 0.
I20260928 02:16:00.219705 27053 leader_election.cc:302] T f71af20203ad4a9fbc9459c6bb1aec7a P 658c6c5db8e34fc8b0c1b608205e7542 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1653d1fcbf534f33a82d04943fa7ed82, 658c6c5db8e34fc8b0c1b608205e7542; no voters:
I20260928 02:16:00.220252 27448 raft_consensus.cc:2781] T f71af20203ad4a9fbc9459c6bb1aec7a P 658c6c5db8e34fc8b0c1b608205e7542 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 02:16:00.220400 27448 raft_consensus.cc:482] T f71af20203ad4a9fbc9459c6bb1aec7a P 658c6c5db8e34fc8b0c1b608205e7542 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 02:16:00.220510 27448 raft_consensus.cc:3037] T f71af20203ad4a9fbc9459c6bb1aec7a P 658c6c5db8e34fc8b0c1b608205e7542 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:16:00.223127 27448 raft_consensus.cc:504] T f71af20203ad4a9fbc9459c6bb1aec7a P 658c6c5db8e34fc8b0c1b608205e7542 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "658c6c5db8e34fc8b0c1b608205e7542" member_type: VOTER last_known_addr { host: "127.6.235.65" port: 40115 } } peers { permanent_uuid: "1653d1fcbf534f33a82d04943fa7ed82" member_type: VOTER last_known_addr { host: "127.6.235.67" port: 44055 } } peers { permanent_uuid: "dfcaab5cc5604949ad13b3cf0c28407e" member_type: VOTER last_known_addr { host: "127.6.235.66" port: 33213 } }
I20260928 02:16:00.223770 27448 leader_election.cc:288] T f71af20203ad4a9fbc9459c6bb1aec7a P 658c6c5db8e34fc8b0c1b608205e7542 [CANDIDATE]: Term 1 election: Requested vote from peers 1653d1fcbf534f33a82d04943fa7ed82 (127.6.235.67:44055), dfcaab5cc5604949ad13b3cf0c28407e (127.6.235.66:33213)
I20260928 02:16:00.224273 27380 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f71af20203ad4a9fbc9459c6bb1aec7a" candidate_uuid: "658c6c5db8e34fc8b0c1b608205e7542" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1653d1fcbf534f33a82d04943fa7ed82"
I20260928 02:16:00.224298 27250 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f71af20203ad4a9fbc9459c6bb1aec7a" candidate_uuid: "658c6c5db8e34fc8b0c1b608205e7542" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dfcaab5cc5604949ad13b3cf0c28407e"
I20260928 02:16:00.224488 27380 raft_consensus.cc:3037] T f71af20203ad4a9fbc9459c6bb1aec7a P 1653d1fcbf534f33a82d04943fa7ed82 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:16:00.224504 27250 raft_consensus.cc:3037] T f71af20203ad4a9fbc9459c6bb1aec7a P dfcaab5cc5604949ad13b3cf0c28407e [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:16:00.226989 27250 raft_consensus.cc:2454] T f71af20203ad4a9fbc9459c6bb1aec7a P dfcaab5cc5604949ad13b3cf0c28407e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 658c6c5db8e34fc8b0c1b608205e7542 in term 1.
I20260928 02:16:00.227123 27380 raft_consensus.cc:2454] T f71af20203ad4a9fbc9459c6bb1aec7a P 1653d1fcbf534f33a82d04943fa7ed82 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 658c6c5db8e34fc8b0c1b608205e7542 in term 1.
I20260928 02:16:00.227491 27055 leader_election.cc:302] T f71af20203ad4a9fbc9459c6bb1aec7a P 658c6c5db8e34fc8b0c1b608205e7542 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 658c6c5db8e34fc8b0c1b608205e7542, dfcaab5cc5604949ad13b3cf0c28407e; no voters:
I20260928 02:16:00.227852 27448 raft_consensus.cc:2781] T f71af20203ad4a9fbc9459c6bb1aec7a P 658c6c5db8e34fc8b0c1b608205e7542 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:16:00.228327 27448 raft_consensus.cc:686] T f71af20203ad4a9fbc9459c6bb1aec7a P 658c6c5db8e34fc8b0c1b608205e7542 [term 1 LEADER]: Becoming Leader. State: Replica: 658c6c5db8e34fc8b0c1b608205e7542, State: Running, Role: LEADER
I20260928 02:16:00.228873 27448 consensus_queue.cc:237] T f71af20203ad4a9fbc9459c6bb1aec7a P 658c6c5db8e34fc8b0c1b608205e7542 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "658c6c5db8e34fc8b0c1b608205e7542" member_type: VOTER last_known_addr { host: "127.6.235.65" port: 40115 } } peers { permanent_uuid: "1653d1fcbf534f33a82d04943fa7ed82" member_type: VOTER last_known_addr { host: "127.6.235.67" port: 44055 } } peers { permanent_uuid: "dfcaab5cc5604949ad13b3cf0c28407e" member_type: VOTER last_known_addr { host: "127.6.235.66" port: 33213 } }
I20260928 02:16:00.233834 26978 catalog_manager.cc:5495] T f71af20203ad4a9fbc9459c6bb1aec7a P 658c6c5db8e34fc8b0c1b608205e7542 reported cstate change: term changed from 0 to 1, leader changed from <none> to 658c6c5db8e34fc8b0c1b608205e7542 (127.6.235.65). New cstate: current_term: 1 leader_uuid: "658c6c5db8e34fc8b0c1b608205e7542" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "658c6c5db8e34fc8b0c1b608205e7542" member_type: VOTER last_known_addr { host: "127.6.235.65" port: 40115 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1653d1fcbf534f33a82d04943fa7ed82" member_type: VOTER last_known_addr { host: "127.6.235.67" port: 44055 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dfcaab5cc5604949ad13b3cf0c28407e" member_type: VOTER last_known_addr { host: "127.6.235.66" port: 33213 } health_report { overall_health: UNKNOWN } } }
I20260928 02:16:00.253338 27438 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:16:00.247922 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50520 (local address 127.6.235.65:40115)
0928 02:16:00.248134 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0928 02:16:00.248139 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 02:16:00.248158 (+ 19us) server_negotiation.cc:407] Connection header received
0928 02:16:00.248351 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:16:00.248355 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:16:00.248446 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:16:00.248631 (+ 185us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 02:16:00.249029 (+ 398us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:16:00.249728 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:16:00.251132 (+ 1404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:16:00.251554 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:16:00.251687 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:16:00.252194 (+ 507us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 02:16:00.252720 (+ 526us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 02:16:00.252884 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0928 02:16:00.252996 (+ 112us) server_negotiation.cc:299] Negotiation successful
0928 02:16:00.253076 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260928 02:16:00.262154 27250 raft_consensus.cc:1261] T f71af20203ad4a9fbc9459c6bb1aec7a P dfcaab5cc5604949ad13b3cf0c28407e [term 1 FOLLOWER]: Refusing update from remote peer 658c6c5db8e34fc8b0c1b608205e7542: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 02:16:00.262153 27380 raft_consensus.cc:1261] T f71af20203ad4a9fbc9459c6bb1aec7a P 1653d1fcbf534f33a82d04943fa7ed82 [term 1 FOLLOWER]: Refusing update from remote peer 658c6c5db8e34fc8b0c1b608205e7542: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 02:16:00.263240 27454 consensus_queue.cc:1035] T f71af20203ad4a9fbc9459c6bb1aec7a P 658c6c5db8e34fc8b0c1b608205e7542 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dfcaab5cc5604949ad13b3cf0c28407e" member_type: VOTER last_known_addr { host: "127.6.235.66" port: 33213 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 02:16:00.263578 27448 consensus_queue.cc:1035] T f71af20203ad4a9fbc9459c6bb1aec7a P 658c6c5db8e34fc8b0c1b608205e7542 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1653d1fcbf534f33a82d04943fa7ed82" member_type: VOTER last_known_addr { host: "127.6.235.67" port: 44055 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 02:16:00.297049 26977 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:55522:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 02:16:00.297361 26977 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:55522:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 02:16:00.301522 26977 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 4c2304e0f7534ec2ba1e48ca8525986f: Sending DeleteTablet for 3 replicas of tablet f71af20203ad4a9fbc9459c6bb1aec7a
I20260928 02:16:00.303014 27360 tablet_service.cc:1514] Processing DeleteTablet for tablet f71af20203ad4a9fbc9459c6bb1aec7a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:16:00 UTC) from {username='slave'} at 127.0.0.1:55808
I20260928 02:16:00.303294 27230 tablet_service.cc:1514] Processing DeleteTablet for tablet f71af20203ad4a9fbc9459c6bb1aec7a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:16:00 UTC) from {username='slave'} at 127.0.0.1:42516
I20260928 02:16:00.303320 27099 tablet_service.cc:1514] Processing DeleteTablet for tablet f71af20203ad4a9fbc9459c6bb1aec7a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:16:00 UTC) from {username='slave'} at 127.0.0.1:50504
I20260928 02:16:00.305280 7085 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskTlaUuc/build/asan/bin/kudu pid 27037
I20260928 02:16:00.334254 27467 tablet_replica.cc:330] T f71af20203ad4a9fbc9459c6bb1aec7a P 1653d1fcbf534f33a82d04943fa7ed82: stopping tablet replica
I20260928 02:16:00.334570 27468 tablet_replica.cc:330] T f71af20203ad4a9fbc9459c6bb1aec7a P dfcaab5cc5604949ad13b3cf0c28407e: stopping tablet replica
I20260928 02:16:00.335111 27468 raft_consensus.cc:2229] T f71af20203ad4a9fbc9459c6bb1aec7a P dfcaab5cc5604949ad13b3cf0c28407e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 02:16:00.335873 27468 raft_consensus.cc:2258] T f71af20203ad4a9fbc9459c6bb1aec7a P dfcaab5cc5604949ad13b3cf0c28407e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 02:16:00.336493 27469 tablet_replica.cc:330] T f71af20203ad4a9fbc9459c6bb1aec7a P 658c6c5db8e34fc8b0c1b608205e7542: stopping tablet replica
I20260928 02:16:00.336961 27469 raft_consensus.cc:2229] T f71af20203ad4a9fbc9459c6bb1aec7a P 658c6c5db8e34fc8b0c1b608205e7542 [term 1 LEADER]: Raft consensus shutting down.
I20260928 02:16:00.337646 27469 raft_consensus.cc:2258] T f71af20203ad4a9fbc9459c6bb1aec7a P 658c6c5db8e34fc8b0c1b608205e7542 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 02:16:00.338127 27467 raft_consensus.cc:2229] T f71af20203ad4a9fbc9459c6bb1aec7a P 1653d1fcbf534f33a82d04943fa7ed82 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 02:16:00.338649 27467 raft_consensus.cc:2258] T f71af20203ad4a9fbc9459c6bb1aec7a P 1653d1fcbf534f33a82d04943fa7ed82 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 02:16:00.339676 27468 ts_tablet_manager.cc:1905] T f71af20203ad4a9fbc9459c6bb1aec7a P dfcaab5cc5604949ad13b3cf0c28407e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 02:16:00.340590 27469 ts_tablet_manager.cc:1905] T f71af20203ad4a9fbc9459c6bb1aec7a P 658c6c5db8e34fc8b0c1b608205e7542: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 02:16:00.340582 27467 ts_tablet_manager.cc:1905] T f71af20203ad4a9fbc9459c6bb1aec7a P 1653d1fcbf534f33a82d04943fa7ed82: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 02:16:00.347725 27468 ts_tablet_manager.cc:1918] T f71af20203ad4a9fbc9459c6bb1aec7a P dfcaab5cc5604949ad13b3cf0c28407e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 02:16:00.348132 27467 ts_tablet_manager.cc:1918] T f71af20203ad4a9fbc9459c6bb1aec7a P 1653d1fcbf534f33a82d04943fa7ed82: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 02:16:00.348192 27468 log.cc:1196] T f71af20203ad4a9fbc9459c6bb1aec7a P dfcaab5cc5604949ad13b3cf0c28407e: Deleting WAL directory at /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790561432251112-7085-0/minicluster-data/ts-1/wal/wals/f71af20203ad4a9fbc9459c6bb1aec7a
I20260928 02:16:00.348326 27467 log.cc:1196] T f71af20203ad4a9fbc9459c6bb1aec7a P 1653d1fcbf534f33a82d04943fa7ed82: Deleting WAL directory at /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790561432251112-7085-0/minicluster-data/ts-2/wal/wals/f71af20203ad4a9fbc9459c6bb1aec7a
I20260928 02:16:00.348883 27467 ts_tablet_manager.cc:1939] T f71af20203ad4a9fbc9459c6bb1aec7a P 1653d1fcbf534f33a82d04943fa7ed82: Deleting consensus metadata
I20260928 02:16:00.349151 27468 ts_tablet_manager.cc:1939] T f71af20203ad4a9fbc9459c6bb1aec7a P dfcaab5cc5604949ad13b3cf0c28407e: Deleting consensus metadata
I20260928 02:16:00.350984 26963 catalog_manager.cc:4841] TS 1653d1fcbf534f33a82d04943fa7ed82 (127.6.235.67:44055): tablet f71af20203ad4a9fbc9459c6bb1aec7a (table test-table [id=4d9e2c5af4304c2d840769a5c1e65d03]) successfully deleted
I20260928 02:16:00.351151 26965 catalog_manager.cc:4841] TS dfcaab5cc5604949ad13b3cf0c28407e (127.6.235.66:33213): tablet f71af20203ad4a9fbc9459c6bb1aec7a (table test-table [id=4d9e2c5af4304c2d840769a5c1e65d03]) successfully deleted
I20260928 02:16:00.351264 27469 ts_tablet_manager.cc:1918] T f71af20203ad4a9fbc9459c6bb1aec7a P 658c6c5db8e34fc8b0c1b608205e7542: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 02:16:00.351583 27469 log.cc:1196] T f71af20203ad4a9fbc9459c6bb1aec7a P 658c6c5db8e34fc8b0c1b608205e7542: Deleting WAL directory at /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790561432251112-7085-0/minicluster-data/ts-0/wal/wals/f71af20203ad4a9fbc9459c6bb1aec7a
I20260928 02:16:00.352067 27469 ts_tablet_manager.cc:1939] T f71af20203ad4a9fbc9459c6bb1aec7a P 658c6c5db8e34fc8b0c1b608205e7542: Deleting consensus metadata
I20260928 02:16:00.353715 26963 catalog_manager.cc:4841] TS 658c6c5db8e34fc8b0c1b608205e7542 (127.6.235.65:40115): tablet f71af20203ad4a9fbc9459c6bb1aec7a (table test-table [id=4d9e2c5af4304c2d840769a5c1e65d03]) successfully deleted
I20260928 02:16:00.355237 27438 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:16:00.306845 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50534 (local address 127.6.235.65:40115)
0928 02:16:00.307082 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0928 02:16:00.307091 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 02:16:00.333076 (+ 25985us) server_negotiation.cc:407] Connection header received
0928 02:16:00.334584 (+ 1508us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:16:00.334592 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:16:00.334726 (+ 134us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:16:00.334811 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:16:00.336109 (+ 1298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:16:00.337416 (+ 1307us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:16:00.345857 (+ 8441us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:16:00.346605 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:16:00.346716 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:16:00.353545 (+ 6829us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:16:00.353556 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:16:00.353575 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 02:16:00.354323 (+ 748us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:16:00.354381 (+ 58us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 02:16:00.354409 (+ 28us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:16:00.354503 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0928 02:16:00.354741 (+ 238us) server_negotiation.cc:299] Negotiation successful
0928 02:16:00.354836 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260928 02:16:00.356122 27159 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 02:16:00.456921 7085 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTlaUuc/build/asan/bin/kudu with pid 27037
I20260928 02:16:00.476982 7085 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskTlaUuc/build/asan/bin/kudu pid 27168
I20260928 02:16:00.484794 27441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:16:00.477877 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42520 (local address 127.6.235.66:33213)
0928 02:16:00.478079 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0928 02:16:00.478086 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 02:16:00.478108 (+ 22us) server_negotiation.cc:407] Connection header received
0928 02:16:00.478170 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:16:00.478176 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:16:00.478269 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:16:00.478391 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:16:00.478770 (+ 379us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:16:00.479969 (+ 1199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:16:00.481353 (+ 1384us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:16:00.482023 (+ 670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:16:00.482145 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:16:00.483197 (+ 1052us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:16:00.483208 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:16:00.483227 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 02:16:00.484023 (+ 796us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:16:00.484084 (+ 61us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 02:16:00.484108 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:16:00.484221 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0928 02:16:00.484323 (+ 102us) server_negotiation.cc:299] Negotiation successful
0928 02:16:00.484435 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260928 02:16:00.485570 27290 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 02:16:00.577584 7085 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTlaUuc/build/asan/bin/kudu with pid 27168
I20260928 02:16:00.595999 7085 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskTlaUuc/build/asan/bin/kudu pid 27298
I20260928 02:16:00.603358 27442 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:16:00.596805 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55824 (local address 127.6.235.67:44055)
0928 02:16:00.597027 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0928 02:16:00.597034 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 02:16:00.597217 (+ 183us) server_negotiation.cc:407] Connection header received
0928 02:16:00.597437 (+ 220us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:16:00.597443 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:16:00.597556 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:16:00.597673 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:16:00.598083 (+ 410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:16:00.598898 (+ 815us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:16:00.600237 (+ 1339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:16:00.600652 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:16:00.600793 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:16:00.601923 (+ 1130us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:16:00.601937 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:16:00.601957 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 02:16:00.602553 (+ 596us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:16:00.602606 (+ 53us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 02:16:00.602627 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:16:00.602788 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0928 02:16:00.602936 (+ 148us) server_negotiation.cc:299] Negotiation successful
0928 02:16:00.603057 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260928 02:16:00.604175 27420 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 02:16:00.698185 7085 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTlaUuc/build/asan/bin/kudu with pid 27298
I20260928 02:16:00.715355 7085 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskTlaUuc/build/asan/bin/kudu pid 26947
I20260928 02:16:00.716574 27008 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 02:16:00.828249 7085 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTlaUuc/build/asan/bin/kudu with pid 26947
2026-09-28T02:16:00Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/5 (2888 ms)
[----------] 6 tests from EncryptionPolicyTest (22660 ms total)
[----------] 18 tests from AuthTokenIssuingTest
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/0
Loading random data
Initializing database '/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790561432251112-7085-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 02:16:00 dist-test-slave-86gv krb5kdc[27477](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 02:16:00 dist-test-slave-86gv krb5kdc[27477](info): set up 2 sockets
Sep 28 02:16:00 dist-test-slave-86gv krb5kdc[27477](info): commencing operation
krb5kdc: starting...
W20260928 02:16:02.018051 7085 mini_kdc.cc:121] Time spent starting KDC: real 1.146s user 0.000s sys 0.030s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790561432251112-7085-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790561432251112-7085-0/krb5kdc/test-user.keytab.
Sep 28 02:16:02 dist-test-slave-86gv krb5kdc[27477](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561762, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T02:16:02Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T02:16:02Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790561432251112-7085-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790561432251112-7085-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 02:16:02.305053 7085 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTlaUuc/build/asan/bin/kudu
/tmp/dist-test-taskTlaUuc/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790561432251112-7085-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790561432251112-7085-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790561432251112-7085-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790561432251112-7085-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45137
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790561432251112-7085-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:35183
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:45137
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790561432251112-7085-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790561432251112-7085-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790561432251112-7085-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:16:02.660167 27490 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:16:02.660574 27490 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:16:02.660737 27490 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:16:02.668503 27490 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 02:16:02.668695 27490 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:16:02.668809 27490 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:16:02.668927 27490 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 02:16:02.668995 27490 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 02:16:02.679631 27490 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:35183
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790561432251112-7085-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790561432251112-7085-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:45137
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790561432251112-7085-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45137
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790561432251112-7085-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790561432251112-7085-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 00:51:59 UTC on ccycloud.quasar-axgkwn.root.comops.site
build id 4676464
ASAN enabled
I20260928 02:16:02.681164 27490 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:16:02.682711 27490 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:16:02.696480 27498 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:16:02.696609 27496 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:16:02.696822 27495 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:16:02.697762 27490 server_base.cc:1034] running on GCE node
I20260928 02:16:02.698706 27490 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:16:02.700434 27490 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:16:02.701700 27490 hybrid_clock.cc:648] HybridClock initialized: now 1790561762701656 us; error 48 us; skew 500 ppm
I20260928 02:16:02.702113 27490 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 02:16:02 dist-test-slave-86gv krb5kdc[27477](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561762, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:16:02.705572 27490 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 02:16:02.707243 27490 webserver.cc:460] Webserver started at http://127.0.0.1:39099/ using document root <none> and password file <none>
I20260928 02:16:02.708003 27490 fs_manager.cc:359] Metadata directory not provided
I20260928 02:16:02.708161 27490 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:16:02.708508 27490 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:16:02.711791 27490 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790561432251112-7085-0/minicluster-data/master-0/data/instance:
uuid: "88d8c357e1884efcaf49a8f438e7dcea"
format_stamp: "Formatted at 2026-09-28 02:16:02 on dist-test-slave-86gv"
server_key: "58964c83843b869dd8d3e8955f29f4ff"
server_key_iv: "a9c305d9af63cedbc8e3bea1b7523270"
server_key_version: "encryptionkey@0"
I20260928 02:16:02.712652 27490 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790561432251112-7085-0/minicluster-data/master-0/wal/instance:
uuid: "88d8c357e1884efcaf49a8f438e7dcea"
format_stamp: "Formatted at 2026-09-28 02:16:02 on dist-test-slave-86gv"
server_key: "58964c83843b869dd8d3e8955f29f4ff"
server_key_iv: "a9c305d9af63cedbc8e3bea1b7523270"
server_key_version: "encryptionkey@0"
I20260928 02:16:02.718590 27490 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260928 02:16:02.722790 27505 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:16:02.724380 27490 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260928 02:16:02.724628 27490 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790561432251112-7085-0/minicluster-data/master-0/data,/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790561432251112-7085-0/minicluster-data/master-0/wal
uuid: "88d8c357e1884efcaf49a8f438e7dcea"
format_stamp: "Formatted at 2026-09-28 02:16:02 on dist-test-slave-86gv"
server_key: "58964c83843b869dd8d3e8955f29f4ff"
server_key_iv: "a9c305d9af63cedbc8e3bea1b7523270"
server_key_version: "encryptionkey@0"
I20260928 02:16:02.724870 27490 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790561432251112-7085-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790561432251112-7085-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790561432251112-7085-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:16:02.745815 27490 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:16:02.750952 27490 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:16:02.751288 27490 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:16:02.771273 27490 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:45137
I20260928 02:16:02.771281 27556 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:45137 every 8 connection(s)
I20260928 02:16:02.772838 27490 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790561432251112-7085-0/minicluster-data/master-0/data/info.pb
I20260928 02:16:02.773329 7085 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTlaUuc/build/asan/bin/kudu as pid 27490
I20260928 02:16:02.773562 7085 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790561432251112-7085-0/minicluster-data/master-0/wal/instance
I20260928 02:16:02.774417 7085 external_mini_cluster.cc:1428] Setting key 72bc66a9ae11acb7f2f9c2bf7503ded5
I20260928 02:16:02.777612 27557 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 28 02:16:02 dist-test-slave-86gv krb5kdc[27477](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561762, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260928 02:16:02.791586 27557 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 88d8c357e1884efcaf49a8f438e7dcea: Bootstrap starting.
I20260928 02:16:02.795111 27557 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 88d8c357e1884efcaf49a8f438e7dcea: Neither blocks nor log segments found. Creating new log.
I20260928 02:16:02.796300 27557 log.cc:824] T 00000000000000000000000000000000 P 88d8c357e1884efcaf49a8f438e7dcea: Log is configured to *not* fsync() on all Append() calls
I20260928 02:16:02.799566 27560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:16:02.776571 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48209 (local address 127.0.0.1:45137)
0928 02:16:02.777406 (+ 835us) server_negotiation.cc:206] Beginning negotiation
0928 02:16:02.777418 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0928 02:16:02.777462 (+ 44us) server_negotiation.cc:407] Connection header received
0928 02:16:02.779181 (+ 1719us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:16:02.779204 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:16:02.779610 (+ 406us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:16:02.779952 (+ 342us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:16:02.781839 (+ 1887us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:16:02.783253 (+ 1414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:16:02.785173 (+ 1920us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:16:02.785940 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:16:02.786117 (+ 177us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:16:02.791247 (+ 5130us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:16:02.791256 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:16:02.791273 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:16:02.791316 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:16:02.794597 (+ 3281us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:16:02.795591 (+ 994us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:16:02.795598 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:16:02.795606 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:16:02.795784 (+ 178us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:16:02.796293 (+ 509us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:16:02.796297 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:16:02.796299 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:16:02.797018 (+ 719us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:16:02.797653 (+ 635us) server_negotiation.cc:1035] Waiting for connection context
0928 02:16:02.798100 (+ 447us) server_negotiation.cc:299] Negotiation successful
0928 02:16:02.798472 (+ 372us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":613,"thread_start_us":405,"threads_started":1}
I20260928 02:16:02.799991 27557 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 88d8c357e1884efcaf49a8f438e7dcea: No bootstrap required, opened a new log
I20260928 02:16:02.804154 7085 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 02:16:02.808698 27557 raft_consensus.cc:348] T 00000000000000000000000000000000 P 88d8c357e1884efcaf49a8f438e7dcea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "88d8c357e1884efcaf49a8f438e7dcea" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45137 } }
I20260928 02:16:02.809283 27557 raft_consensus.cc:374] T 00000000000000000000000000000000 P 88d8c357e1884efcaf49a8f438e7dcea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:16:02.809535 27557 raft_consensus.cc:729] T 00000000000000000000000000000000 P 88d8c357e1884efcaf49a8f438e7dcea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 88d8c357e1884efcaf49a8f438e7dcea, State: Initialized, Role: FOLLOWER
I20260928 02:16:02.810428 27557 consensus_queue.cc:260] T 00000000000000000000000000000000 P 88d8c357e1884efcaf49a8f438e7dcea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "88d8c357e1884efcaf49a8f438e7dcea" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45137 } }
I20260928 02:16:02.810740 27557 raft_consensus.cc:388] T 00000000000000000000000000000000 P 88d8c357e1884efcaf49a8f438e7dcea [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 02:16:02.810899 27557 raft_consensus.cc:482] T 00000000000000000000000000000000 P 88d8c357e1884efcaf49a8f438e7dcea [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 02:16:02.811097 27557 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 88d8c357e1884efcaf49a8f438e7dcea [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:16:02.814250 27557 raft_consensus.cc:504] T 00000000000000000000000000000000 P 88d8c357e1884efcaf49a8f438e7dcea [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "88d8c357e1884efcaf49a8f438e7dcea" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45137 } }
I20260928 02:16:02.814982 27557 leader_election.cc:302] T 00000000000000000000000000000000 P 88d8c357e1884efcaf49a8f438e7dcea [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 88d8c357e1884efcaf49a8f438e7dcea; no voters:
I20260928 02:16:02.815780 27557 leader_election.cc:288] T 00000000000000000000000000000000 P 88d8c357e1884efcaf49a8f438e7dcea [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 02:16:02.815914 27567 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 88d8c357e1884efcaf49a8f438e7dcea [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:16:02.825230 27567 raft_consensus.cc:686] T 00000000000000000000000000000000 P 88d8c357e1884efcaf49a8f438e7dcea [term 1 LEADER]: Becoming Leader. State: Replica: 88d8c357e1884efcaf49a8f438e7dcea, State: Running, Role: LEADER
I20260928 02:16:02.826287 27567 consensus_queue.cc:237] T 00000000000000000000000000000000 P 88d8c357e1884efcaf49a8f438e7dcea [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "88d8c357e1884efcaf49a8f438e7dcea" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45137 } }
I20260928 02:16:02.834100 27557 sys_catalog.cc:564] T 00000000000000000000000000000000 P 88d8c357e1884efcaf49a8f438e7dcea [sys.catalog]: configured and running, proceeding with master startup.
I20260928 02:16:02.836071 27569 sys_catalog.cc:455] T 00000000000000000000000000000000 P 88d8c357e1884efcaf49a8f438e7dcea [sys.catalog]: SysCatalogTable state changed. Reason: New leader 88d8c357e1884efcaf49a8f438e7dcea. Latest consensus state: current_term: 1 leader_uuid: "88d8c357e1884efcaf49a8f438e7dcea" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "88d8c357e1884efcaf49a8f438e7dcea" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45137 } } }
I20260928 02:16:02.836621 27569 sys_catalog.cc:458] T 00000000000000000000000000000000 P 88d8c357e1884efcaf49a8f438e7dcea [sys.catalog]: This master's current role is: LEADER
I20260928 02:16:02.836036 27568 sys_catalog.cc:455] T 00000000000000000000000000000000 P 88d8c357e1884efcaf49a8f438e7dcea [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "88d8c357e1884efcaf49a8f438e7dcea" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "88d8c357e1884efcaf49a8f438e7dcea" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45137 } } }
I20260928 02:16:02.838202 27568 sys_catalog.cc:458] T 00000000000000000000000000000000 P 88d8c357e1884efcaf49a8f438e7dcea [sys.catalog]: This master's current role is: LEADER
I20260928 02:16:02.838502 27560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:16:02.809191 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39117 (local address 127.0.0.1:45137)
0928 02:16:02.809398 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0928 02:16:02.809405 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 02:16:02.809895 (+ 490us) server_negotiation.cc:407] Connection header received
0928 02:16:02.810116 (+ 221us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:16:02.810121 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:16:02.810214 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:16:02.810333 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:16:02.812347 (+ 2014us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:16:02.813637 (+ 1290us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:16:02.815320 (+ 1683us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:16:02.817208 (+ 1888us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:16:02.817432 (+ 224us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:16:02.826874 (+ 9442us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:16:02.826881 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:16:02.826886 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:16:02.826932 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:16:02.833735 (+ 6803us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:16:02.834947 (+ 1212us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:16:02.834953 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:16:02.834956 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:16:02.835217 (+ 261us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:16:02.835888 (+ 671us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:16:02.835893 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:16:02.835896 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:16:02.836349 (+ 453us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:16:02.836674 (+ 325us) server_negotiation.cc:1035] Waiting for connection context
0928 02:16:02.837919 (+ 1245us) server_negotiation.cc:299] Negotiation successful
0928 02:16:02.838145 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260928 02:16:02.841701 27573 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 02:16:02.847488 27573 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 02:16:02.857079 27573 catalog_manager.cc:1347] Generated new cluster ID: 6f66113a70c348709e0213198e06240f
I20260928 02:16:02.857223 27573 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 02:16:02.865150 27573 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 02:16:02.866608 27573 catalog_manager.cc:1504] Loading token signing keys...
I20260928 02:16:02.877911 27573 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 88d8c357e1884efcaf49a8f438e7dcea: Generated new TSK 0
I20260928 02:16:02.879106 27573 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 02:16:02.896117 7085 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskTlaUuc/build/asan/bin/kudu pid 27490
I20260928 02:16:02.897701 27552 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 02:16:02.992223 7085 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTlaUuc/build/asan/bin/kudu with pid 27490
W20260928 02:16:03.010685 27475 connection.cc:535] client connection to 127.0.0.1:45137 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:45137): Connection reset by peer (error 104)
2026-09-28T02:16:03Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/0 (2162 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/1
Loading random data
Initializing database '/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790561432251112-7085-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 02:16:03 dist-test-slave-86gv krb5kdc[27589](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 02:16:03 dist-test-slave-86gv krb5kdc[27589](info): set up 2 sockets
Sep 28 02:16:03 dist-test-slave-86gv krb5kdc[27589](info): commencing operation
krb5kdc: starting...
W20260928 02:16:04.145871 7085 mini_kdc.cc:121] Time spent starting KDC: real 1.112s user 0.004s sys 0.025s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790561432251112-7085-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790561432251112-7085-0/krb5kdc/test-user.keytab.
Sep 28 02:16:04 dist-test-slave-86gv krb5kdc[27589](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561764, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T02:16:04Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T02:16:04Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790561432251112-7085-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790561432251112-7085-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 02:16:04.428395 7085 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTlaUuc/build/asan/bin/kudu
/tmp/dist-test-taskTlaUuc/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790561432251112-7085-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790561432251112-7085-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790561432251112-7085-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790561432251112-7085-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33995
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790561432251112-7085-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:45325
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:33995
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790561432251112-7085-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790561432251112-7085-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790561432251112-7085-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:16:04.790449 27602 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:16:04.790827 27602 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:16:04.791030 27602 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:16:04.798588 27602 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 02:16:04.798738 27602 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:16:04.798852 27602 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:16:04.798951 27602 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 02:16:04.799068 27602 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 02:16:04.809829 27602 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:45325
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790561432251112-7085-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790561432251112-7085-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:33995
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790561432251112-7085-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33995
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790561432251112-7085-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790561432251112-7085-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 00:51:59 UTC on ccycloud.quasar-axgkwn.root.comops.site
build id 4676464
ASAN enabled
I20260928 02:16:04.811370 27602 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:16:04.812979 27602 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:16:04.826277 27607 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:16:04.826396 27608 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:16:04.826263 27610 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:16:04.827409 27602 server_base.cc:1034] running on GCE node
I20260928 02:16:04.828297 27602 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:16:04.829893 27602 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:16:04.831112 27602 hybrid_clock.cc:648] HybridClock initialized: now 1790561764831072 us; error 31 us; skew 500 ppm
I20260928 02:16:04.831439 27602 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 02:16:04 dist-test-slave-86gv krb5kdc[27589](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561764, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:16:04.834896 27602 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 02:16:04.836426 27602 webserver.cc:460] Webserver started at http://127.0.0.1:41931/ using document root <none> and password file <none>
I20260928 02:16:04.837281 27602 fs_manager.cc:359] Metadata directory not provided
I20260928 02:16:04.837393 27602 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:16:04.837656 27602 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:16:04.840716 27602 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790561432251112-7085-0/minicluster-data/master-0/data/instance:
uuid: "c43f6f245da84b7193a3be3aece4255c"
format_stamp: "Formatted at 2026-09-28 02:16:04 on dist-test-slave-86gv"
server_key: "b6aa4baf957c1509096029d58b6c9db8"
server_key_iv: "6cee3e9c8df2819de187d6bcb44eab45"
server_key_version: "encryptionkey@0"
I20260928 02:16:04.841603 27602 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790561432251112-7085-0/minicluster-data/master-0/wal/instance:
uuid: "c43f6f245da84b7193a3be3aece4255c"
format_stamp: "Formatted at 2026-09-28 02:16:04 on dist-test-slave-86gv"
server_key: "b6aa4baf957c1509096029d58b6c9db8"
server_key_iv: "6cee3e9c8df2819de187d6bcb44eab45"
server_key_version: "encryptionkey@0"
I20260928 02:16:04.847476 27602 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.001s sys 0.004s
I20260928 02:16:04.851547 27617 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:16:04.853303 27602 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260928 02:16:04.853461 27602 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790561432251112-7085-0/minicluster-data/master-0/data,/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790561432251112-7085-0/minicluster-data/master-0/wal
uuid: "c43f6f245da84b7193a3be3aece4255c"
format_stamp: "Formatted at 2026-09-28 02:16:04 on dist-test-slave-86gv"
server_key: "b6aa4baf957c1509096029d58b6c9db8"
server_key_iv: "6cee3e9c8df2819de187d6bcb44eab45"
server_key_version: "encryptionkey@0"
I20260928 02:16:04.853652 27602 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790561432251112-7085-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790561432251112-7085-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790561432251112-7085-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:16:04.892004 27602 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:16:04.896807 27602 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:16:04.897141 27602 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:16:04.917161 27602 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:33995
I20260928 02:16:04.917233 27668 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:33995 every 8 connection(s)
I20260928 02:16:04.918613 27602 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790561432251112-7085-0/minicluster-data/master-0/data/info.pb
I20260928 02:16:04.922822 27669 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:16:04.927048 7085 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTlaUuc/build/asan/bin/kudu as pid 27602
I20260928 02:16:04.927235 7085 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790561432251112-7085-0/minicluster-data/master-0/wal/instance
I20260928 02:16:04.927670 7085 external_mini_cluster.cc:1428] Setting key 9c806185bf563f23234a03ffa146b792
I20260928 02:16:04.936031 27669 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c43f6f245da84b7193a3be3aece4255c: Bootstrap starting.
I20260928 02:16:04.939831 27669 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c43f6f245da84b7193a3be3aece4255c: Neither blocks nor log segments found. Creating new log.
I20260928 02:16:04.941219 27669 log.cc:824] T 00000000000000000000000000000000 P c43f6f245da84b7193a3be3aece4255c: Log is configured to *not* fsync() on all Append() calls
I20260928 02:16:04.941630 27672 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:16:04.929662 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54309 (local address 127.0.0.1:33995)
0928 02:16:04.930440 (+ 778us) server_negotiation.cc:206] Beginning negotiation
0928 02:16:04.930453 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0928 02:16:04.930494 (+ 41us) server_negotiation.cc:407] Connection header received
0928 02:16:04.932215 (+ 1721us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:16:04.932240 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:16:04.932746 (+ 506us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:16:04.933249 (+ 503us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:16:04.933837 (+ 588us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:16:04.935163 (+ 1326us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:16:04.936581 (+ 1418us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:16:04.937332 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:16:04.937549 (+ 217us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:16:04.938973 (+ 1424us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:16:04.938980 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:16:04.938994 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 02:16:04.939576 (+ 582us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:16:04.939651 (+ 75us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 02:16:04.939676 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:16:04.939881 (+ 205us) server_negotiation.cc:1035] Waiting for connection context
0928 02:16:04.940358 (+ 477us) server_negotiation.cc:299] Negotiation successful
0928 02:16:04.940512 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":569,"thread_start_us":338,"threads_started":1}
I20260928 02:16:04.944665 27669 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c43f6f245da84b7193a3be3aece4255c: No bootstrap required, opened a new log
I20260928 02:16:04.946139 7085 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 02:16:04.953354 27669 raft_consensus.cc:348] T 00000000000000000000000000000000 P c43f6f245da84b7193a3be3aece4255c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c43f6f245da84b7193a3be3aece4255c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33995 } }
I20260928 02:16:04.953840 27669 raft_consensus.cc:374] T 00000000000000000000000000000000 P c43f6f245da84b7193a3be3aece4255c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:16:04.953966 27669 raft_consensus.cc:729] T 00000000000000000000000000000000 P c43f6f245da84b7193a3be3aece4255c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c43f6f245da84b7193a3be3aece4255c, State: Initialized, Role: FOLLOWER
I20260928 02:16:04.954730 27669 consensus_queue.cc:260] T 00000000000000000000000000000000 P c43f6f245da84b7193a3be3aece4255c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c43f6f245da84b7193a3be3aece4255c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33995 } }
I20260928 02:16:04.955015 27669 raft_consensus.cc:388] T 00000000000000000000000000000000 P c43f6f245da84b7193a3be3aece4255c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 02:16:04.955197 27669 raft_consensus.cc:482] T 00000000000000000000000000000000 P c43f6f245da84b7193a3be3aece4255c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 02:16:04.955410 27669 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c43f6f245da84b7193a3be3aece4255c [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:16:04.958503 27672 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:16:04.951056 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33117 (local address 127.0.0.1:33995)
0928 02:16:04.951291 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0928 02:16:04.951299 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 02:16:04.951625 (+ 326us) server_negotiation.cc:407] Connection header received
0928 02:16:04.951871 (+ 246us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:16:04.951876 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:16:04.951950 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:16:04.952077 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:16:04.952624 (+ 547us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:16:04.953512 (+ 888us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:16:04.955281 (+ 1769us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:16:04.955854 (+ 573us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:16:04.955969 (+ 115us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:16:04.957501 (+ 1532us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:16:04.957508 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:16:04.957513 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 02:16:04.957541 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:16:04.957612 (+ 71us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 02:16:04.957632 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:16:04.957731 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0928 02:16:04.958015 (+ 284us) server_negotiation.cc:299] Negotiation successful
0928 02:16:04.958091 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260928 02:16:04.959174 27669 raft_consensus.cc:504] T 00000000000000000000000000000000 P c43f6f245da84b7193a3be3aece4255c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c43f6f245da84b7193a3be3aece4255c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33995 } }
I20260928 02:16:04.959779 27669 leader_election.cc:302] T 00000000000000000000000000000000 P c43f6f245da84b7193a3be3aece4255c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c43f6f245da84b7193a3be3aece4255c; no voters:
I20260928 02:16:04.960508 27669 leader_election.cc:288] T 00000000000000000000000000000000 P c43f6f245da84b7193a3be3aece4255c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 02:16:04.960628 27679 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c43f6f245da84b7193a3be3aece4255c [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:16:04.961393 27679 raft_consensus.cc:686] T 00000000000000000000000000000000 P c43f6f245da84b7193a3be3aece4255c [term 1 LEADER]: Becoming Leader. State: Replica: c43f6f245da84b7193a3be3aece4255c, State: Running, Role: LEADER
I20260928 02:16:04.962159 27679 consensus_queue.cc:237] T 00000000000000000000000000000000 P c43f6f245da84b7193a3be3aece4255c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c43f6f245da84b7193a3be3aece4255c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33995 } }
I20260928 02:16:04.963258 27669 sys_catalog.cc:564] T 00000000000000000000000000000000 P c43f6f245da84b7193a3be3aece4255c [sys.catalog]: configured and running, proceeding with master startup.
I20260928 02:16:04.967046 27680 sys_catalog.cc:455] T 00000000000000000000000000000000 P c43f6f245da84b7193a3be3aece4255c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c43f6f245da84b7193a3be3aece4255c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c43f6f245da84b7193a3be3aece4255c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33995 } } }
I20260928 02:16:04.967489 27680 sys_catalog.cc:458] T 00000000000000000000000000000000 P c43f6f245da84b7193a3be3aece4255c [sys.catalog]: This master's current role is: LEADER
I20260928 02:16:04.967144 27681 sys_catalog.cc:455] T 00000000000000000000000000000000 P c43f6f245da84b7193a3be3aece4255c [sys.catalog]: SysCatalogTable state changed. Reason: New leader c43f6f245da84b7193a3be3aece4255c. Latest consensus state: current_term: 1 leader_uuid: "c43f6f245da84b7193a3be3aece4255c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c43f6f245da84b7193a3be3aece4255c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33995 } } }
I20260928 02:16:04.967615 27681 sys_catalog.cc:458] T 00000000000000000000000000000000 P c43f6f245da84b7193a3be3aece4255c [sys.catalog]: This master's current role is: LEADER
I20260928 02:16:04.968750 27688 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 02:16:04.983309 27688 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 02:16:04.996417 27688 catalog_manager.cc:1347] Generated new cluster ID: e3f61c97fcb24ec0923ccb27c73af6db
I20260928 02:16:04.996556 27688 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 02:16:05.012847 27688 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 02:16:05.014556 27688 catalog_manager.cc:1504] Loading token signing keys...
I20260928 02:16:05.023126 27688 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c43f6f245da84b7193a3be3aece4255c: Generated new TSK 0
I20260928 02:16:05.024160 27688 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 02:16:05.038206 7085 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskTlaUuc/build/asan/bin/kudu pid 27602
W20260928 02:16:05.039425 27664 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:54309
W20260928 02:16:05.040359 7085 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskTlaUuc/build/asan/bin/kudu pid 27602: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260928 02:16:05.040548 7085 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTlaUuc/build/asan/bin/kudu with pid 27602
W20260928 02:16:05.058877 27587 connection.cc:535] client connection to 127.0.0.1:33995 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:33995): Connection reset by peer (error 104)
2026-09-28T02:16:05Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/1 (2045 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/2
Loading random data
Initializing database '/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790561432251112-7085-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 02:16:05 dist-test-slave-86gv krb5kdc[27700](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 02:16:05 dist-test-slave-86gv krb5kdc[27700](info): set up 2 sockets
Sep 28 02:16:05 dist-test-slave-86gv krb5kdc[27700](info): commencing operation
krb5kdc: starting...
W20260928 02:16:06.165759 7085 mini_kdc.cc:121] Time spent starting KDC: real 1.086s user 0.004s sys 0.025s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790561432251112-7085-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790561432251112-7085-0/krb5kdc/test-user.keytab.
Sep 28 02:16:06 dist-test-slave-86gv krb5kdc[27700](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561766, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T02:16:06Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T02:16:06Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790561432251112-7085-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790561432251112-7085-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 02:16:06.444484 7085 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTlaUuc/build/asan/bin/kudu
/tmp/dist-test-taskTlaUuc/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790561432251112-7085-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790561432251112-7085-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790561432251112-7085-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790561432251112-7085-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35181
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790561432251112-7085-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:35719
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:35181
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790561432251112-7085-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790561432251112-7085-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790561432251112-7085-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:16:06.792800 27713 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:16:06.793138 27713 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:16:06.793341 27713 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:16:06.800920 27713 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 02:16:06.801055 27713 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:16:06.801199 27713 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:16:06.801302 27713 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 02:16:06.801389 27713 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 02:16:06.811733 27713 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:35719
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790561432251112-7085-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790561432251112-7085-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:35181
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790561432251112-7085-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35181
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790561432251112-7085-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790561432251112-7085-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 00:51:59 UTC on ccycloud.quasar-axgkwn.root.comops.site
build id 4676464
ASAN enabled
I20260928 02:16:06.813252 27713 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:16:06.814644 27713 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:16:06.827586 27718 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:16:06.827615 27721 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:16:06.827558 27719 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:16:06.828524 27713 server_base.cc:1034] running on GCE node
I20260928 02:16:06.829409 27713 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:16:06.831039 27713 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:16:06.832472 27713 hybrid_clock.cc:648] HybridClock initialized: now 1790561766832409 us; error 51 us; skew 500 ppm
I20260928 02:16:06.833010 27713 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 02:16:06 dist-test-slave-86gv krb5kdc[27700](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561766, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:16:06.841686 27713 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 02:16:06.843336 27713 webserver.cc:460] Webserver started at http://127.0.0.1:37059/ using document root <none> and password file <none>
I20260928 02:16:06.844097 27713 fs_manager.cc:359] Metadata directory not provided
I20260928 02:16:06.844252 27713 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:16:06.844580 27713 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:16:06.847770 27713 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790561432251112-7085-0/minicluster-data/master-0/data/instance:
uuid: "d6865207f95340fe9d4a7440e1593958"
format_stamp: "Formatted at 2026-09-28 02:16:06 on dist-test-slave-86gv"
server_key: "9ef704ebdb702eabecec60f032e3bc01"
server_key_iv: "acc876f62b8c4683d98cc81a9f885551"
server_key_version: "encryptionkey@0"
I20260928 02:16:06.848611 27713 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790561432251112-7085-0/minicluster-data/master-0/wal/instance:
uuid: "d6865207f95340fe9d4a7440e1593958"
format_stamp: "Formatted at 2026-09-28 02:16:06 on dist-test-slave-86gv"
server_key: "9ef704ebdb702eabecec60f032e3bc01"
server_key_iv: "acc876f62b8c4683d98cc81a9f885551"
server_key_version: "encryptionkey@0"
I20260928 02:16:06.854303 27713 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.002s
I20260928 02:16:06.858223 27728 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:16:06.859736 27713 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260928 02:16:06.859946 27713 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790561432251112-7085-0/minicluster-data/master-0/data,/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790561432251112-7085-0/minicluster-data/master-0/wal
uuid: "d6865207f95340fe9d4a7440e1593958"
format_stamp: "Formatted at 2026-09-28 02:16:06 on dist-test-slave-86gv"
server_key: "9ef704ebdb702eabecec60f032e3bc01"
server_key_iv: "acc876f62b8c4683d98cc81a9f885551"
server_key_version: "encryptionkey@0"
I20260928 02:16:06.860167 27713 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790561432251112-7085-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790561432251112-7085-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790561432251112-7085-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:16:06.866426 27713 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:16:06.870870 27713 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:16:06.871183 27713 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:16:06.890421 27713 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:35181
I20260928 02:16:06.890468 27779 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:35181 every 8 connection(s)
I20260928 02:16:06.892077 27713 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790561432251112-7085-0/minicluster-data/master-0/data/info.pb
I20260928 02:16:06.896337 27780 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:16:06.901656 7085 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTlaUuc/build/asan/bin/kudu as pid 27713
I20260928 02:16:06.901896 7085 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790561432251112-7085-0/minicluster-data/master-0/wal/instance
I20260928 02:16:06.902375 7085 external_mini_cluster.cc:1428] Setting key b4dd2ec1f15a0481c6c64ada18c9962b
I20260928 02:16:06.907809 27780 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d6865207f95340fe9d4a7440e1593958: Bootstrap starting.
I20260928 02:16:06.911235 27783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:16:06.904538 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60075 (local address 127.0.0.1:35181)
0928 02:16:06.905663 (+ 1125us) server_negotiation.cc:206] Beginning negotiation
0928 02:16:06.905676 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0928 02:16:06.905729 (+ 53us) server_negotiation.cc:407] Connection header received
0928 02:16:06.907492 (+ 1763us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:16:06.907517 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:16:06.907913 (+ 396us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:16:06.908326 (+ 413us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:16:06.909522 (+ 1196us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:16:06.909529 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:16:06.909546 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 02:16:06.909562 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:16:06.909640 (+ 78us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 02:16:06.909672 (+ 32us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:16:06.909867 (+ 195us) server_negotiation.cc:1035] Waiting for connection context
0928 02:16:06.909977 (+ 110us) server_negotiation.cc:299] Negotiation successful
0928 02:16:06.910199 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":861,"thread_start_us":582,"threads_started":1}
I20260928 02:16:06.911728 27780 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d6865207f95340fe9d4a7440e1593958: Neither blocks nor log segments found. Creating new log.
I20260928 02:16:06.912812 27780 log.cc:824] T 00000000000000000000000000000000 P d6865207f95340fe9d4a7440e1593958: Log is configured to *not* fsync() on all Append() calls
I20260928 02:16:06.915369 7085 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 02:16:06.916146 27780 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d6865207f95340fe9d4a7440e1593958: No bootstrap required, opened a new log
I20260928 02:16:06.923307 27783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:16:06.920559 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59535 (local address 127.0.0.1:35181)
0928 02:16:06.920796 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0928 02:16:06.920805 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 02:16:06.921257 (+ 452us) server_negotiation.cc:407] Connection header received
0928 02:16:06.921441 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:16:06.921445 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:16:06.921502 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:16:06.921590 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:16:06.922591 (+ 1001us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:16:06.922596 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:16:06.922599 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 02:16:06.922617 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:16:06.922656 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 02:16:06.922671 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:16:06.922775 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0928 02:16:06.922924 (+ 149us) server_negotiation.cc:299] Negotiation successful
0928 02:16:06.922978 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260928 02:16:06.926071 27780 raft_consensus.cc:348] T 00000000000000000000000000000000 P d6865207f95340fe9d4a7440e1593958 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d6865207f95340fe9d4a7440e1593958" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35181 } }
I20260928 02:16:06.926510 27780 raft_consensus.cc:374] T 00000000000000000000000000000000 P d6865207f95340fe9d4a7440e1593958 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:16:06.926633 27780 raft_consensus.cc:729] T 00000000000000000000000000000000 P d6865207f95340fe9d4a7440e1593958 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d6865207f95340fe9d4a7440e1593958, State: Initialized, Role: FOLLOWER
I20260928 02:16:06.927241 27780 consensus_queue.cc:260] T 00000000000000000000000000000000 P d6865207f95340fe9d4a7440e1593958 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d6865207f95340fe9d4a7440e1593958" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35181 } }
I20260928 02:16:06.927505 27780 raft_consensus.cc:388] T 00000000000000000000000000000000 P d6865207f95340fe9d4a7440e1593958 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 02:16:06.927651 27780 raft_consensus.cc:482] T 00000000000000000000000000000000 P d6865207f95340fe9d4a7440e1593958 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 02:16:06.927827 27780 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d6865207f95340fe9d4a7440e1593958 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:16:06.930238 27780 raft_consensus.cc:504] T 00000000000000000000000000000000 P d6865207f95340fe9d4a7440e1593958 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d6865207f95340fe9d4a7440e1593958" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35181 } }
I20260928 02:16:06.930814 27780 leader_election.cc:302] T 00000000000000000000000000000000 P d6865207f95340fe9d4a7440e1593958 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d6865207f95340fe9d4a7440e1593958; no voters:
I20260928 02:16:06.931407 27780 leader_election.cc:288] T 00000000000000000000000000000000 P d6865207f95340fe9d4a7440e1593958 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 02:16:06.931567 27790 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d6865207f95340fe9d4a7440e1593958 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:16:06.932289 27790 raft_consensus.cc:686] T 00000000000000000000000000000000 P d6865207f95340fe9d4a7440e1593958 [term 1 LEADER]: Becoming Leader. State: Replica: d6865207f95340fe9d4a7440e1593958, State: Running, Role: LEADER
I20260928 02:16:06.933034 27790 consensus_queue.cc:237] T 00000000000000000000000000000000 P d6865207f95340fe9d4a7440e1593958 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d6865207f95340fe9d4a7440e1593958" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35181 } }
I20260928 02:16:06.933689 27780 sys_catalog.cc:564] T 00000000000000000000000000000000 P d6865207f95340fe9d4a7440e1593958 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 02:16:06.936228 27792 sys_catalog.cc:455] T 00000000000000000000000000000000 P d6865207f95340fe9d4a7440e1593958 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d6865207f95340fe9d4a7440e1593958. Latest consensus state: current_term: 1 leader_uuid: "d6865207f95340fe9d4a7440e1593958" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d6865207f95340fe9d4a7440e1593958" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35181 } } }
I20260928 02:16:06.936313 27791 sys_catalog.cc:455] T 00000000000000000000000000000000 P d6865207f95340fe9d4a7440e1593958 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d6865207f95340fe9d4a7440e1593958" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d6865207f95340fe9d4a7440e1593958" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35181 } } }
I20260928 02:16:06.936654 27792 sys_catalog.cc:458] T 00000000000000000000000000000000 P d6865207f95340fe9d4a7440e1593958 [sys.catalog]: This master's current role is: LEADER
I20260928 02:16:06.936733 27791 sys_catalog.cc:458] T 00000000000000000000000000000000 P d6865207f95340fe9d4a7440e1593958 [sys.catalog]: This master's current role is: LEADER
I20260928 02:16:06.937645 27798 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 02:16:06.944124 27798 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 02:16:06.953281 27798 catalog_manager.cc:1347] Generated new cluster ID: 454981e452ee423283da7015eff0d244
I20260928 02:16:06.953424 27798 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 02:16:06.964570 27798 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 02:16:06.965121 27798 catalog_manager.cc:1504] Loading token signing keys...
I20260928 02:16:06.973014 27798 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d6865207f95340fe9d4a7440e1593958: Generated new TSK 0
I20260928 02:16:06.974278 27798 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 02:16:06.979557 7085 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskTlaUuc/build/asan/bin/kudu pid 27713
W20260928 02:16:06.981041 27775 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:60075
W20260928 02:16:06.981755 7085 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskTlaUuc/build/asan/bin/kudu pid 27713: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260928 02:16:06.981905 7085 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTlaUuc/build/asan/bin/kudu with pid 27713
2026-09-28T02:16:06Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/2 (1940 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/3
Loading random data
Initializing database '/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790561432251112-7085-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 02:16:07 dist-test-slave-86gv krb5kdc[27811](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 02:16:07 dist-test-slave-86gv krb5kdc[27811](info): set up 2 sockets
Sep 28 02:16:07 dist-test-slave-86gv krb5kdc[27811](info): commencing operation
krb5kdc: starting...
W20260928 02:16:08.149796 7085 mini_kdc.cc:121] Time spent starting KDC: real 1.131s user 0.000s sys 0.029s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790561432251112-7085-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790561432251112-7085-0/krb5kdc/test-user.keytab.
Sep 28 02:16:08 dist-test-slave-86gv krb5kdc[27811](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561768, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T02:16:08Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T02:16:08Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790561432251112-7085-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790561432251112-7085-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 02:16:08.425403 7085 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTlaUuc/build/asan/bin/kudu
/tmp/dist-test-taskTlaUuc/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790561432251112-7085-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790561432251112-7085-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790561432251112-7085-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790561432251112-7085-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41917
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790561432251112-7085-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:35821
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:41917
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790561432251112-7085-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790561432251112-7085-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790561432251112-7085-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:16:08.775828 27824 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:16:08.776129 27824 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:16:08.776309 27824 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:16:08.783949 27824 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 02:16:08.784067 27824 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:16:08.784173 27824 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:16:08.784279 27824 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 02:16:08.784363 27824 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 02:16:08.794787 27824 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:35821
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790561432251112-7085-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790561432251112-7085-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:41917
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790561432251112-7085-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41917
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790561432251112-7085-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790561432251112-7085-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 00:51:59 UTC on ccycloud.quasar-axgkwn.root.comops.site
build id 4676464
ASAN enabled
I20260928 02:16:08.796192 27824 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:16:08.797652 27824 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:16:08.810341 27829 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:16:08.810448 27830 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:16:08.810443 27832 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:16:08.811501 27824 server_base.cc:1034] running on GCE node
I20260928 02:16:08.812420 27824 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:16:08.814388 27824 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:16:08.815650 27824 hybrid_clock.cc:648] HybridClock initialized: now 1790561768815610 us; error 34 us; skew 500 ppm
I20260928 02:16:08.816179 27824 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 02:16:08 dist-test-slave-86gv krb5kdc[27811](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561768, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:16:08.819766 27824 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 02:16:08.821465 27824 webserver.cc:460] Webserver started at http://127.0.0.1:37809/ using document root <none> and password file <none>
I20260928 02:16:08.822178 27824 fs_manager.cc:359] Metadata directory not provided
I20260928 02:16:08.822278 27824 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:16:08.822539 27824 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:16:08.825747 27824 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790561432251112-7085-0/minicluster-data/master-0/data/instance:
uuid: "3fc89e191fbc482eacaa67e1b4deba42"
format_stamp: "Formatted at 2026-09-28 02:16:08 on dist-test-slave-86gv"
server_key: "9ce6cefe5a391032966a55dce0bd6ea9"
server_key_iv: "d5ba8b1b445de88d0ec1966a604ce0af"
server_key_version: "encryptionkey@0"
I20260928 02:16:08.826589 27824 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790561432251112-7085-0/minicluster-data/master-0/wal/instance:
uuid: "3fc89e191fbc482eacaa67e1b4deba42"
format_stamp: "Formatted at 2026-09-28 02:16:08 on dist-test-slave-86gv"
server_key: "9ce6cefe5a391032966a55dce0bd6ea9"
server_key_iv: "d5ba8b1b445de88d0ec1966a604ce0af"
server_key_version: "encryptionkey@0"
I20260928 02:16:08.832517 27824 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.000s
I20260928 02:16:08.836311 27839 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:16:08.838013 27824 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260928 02:16:08.838225 27824 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790561432251112-7085-0/minicluster-data/master-0/data,/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790561432251112-7085-0/minicluster-data/master-0/wal
uuid: "3fc89e191fbc482eacaa67e1b4deba42"
format_stamp: "Formatted at 2026-09-28 02:16:08 on dist-test-slave-86gv"
server_key: "9ce6cefe5a391032966a55dce0bd6ea9"
server_key_iv: "d5ba8b1b445de88d0ec1966a604ce0af"
server_key_version: "encryptionkey@0"
I20260928 02:16:08.838449 27824 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790561432251112-7085-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790561432251112-7085-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790561432251112-7085-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:16:08.853125 27824 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:16:08.858004 27824 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:16:08.858372 27824 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:16:08.878356 27824 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:41917
I20260928 02:16:08.878386 27890 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:41917 every 8 connection(s)
I20260928 02:16:08.879881 27824 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790561432251112-7085-0/minicluster-data/master-0/data/info.pb
I20260928 02:16:08.881865 7085 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTlaUuc/build/asan/bin/kudu as pid 27824
I20260928 02:16:08.882062 7085 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790561432251112-7085-0/minicluster-data/master-0/wal/instance
I20260928 02:16:08.882602 7085 external_mini_cluster.cc:1428] Setting key b6cce4d470133a18bc407ff6ca974483
I20260928 02:16:08.884935 27891 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 28 02:16:08 dist-test-slave-86gv krb5kdc[27811](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561768, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260928 02:16:08.899469 27891 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3fc89e191fbc482eacaa67e1b4deba42: Bootstrap starting.
I20260928 02:16:08.903995 27891 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3fc89e191fbc482eacaa67e1b4deba42: Neither blocks nor log segments found. Creating new log.
I20260928 02:16:08.905350 27891 log.cc:824] T 00000000000000000000000000000000 P 3fc89e191fbc482eacaa67e1b4deba42: Log is configured to *not* fsync() on all Append() calls
I20260928 02:16:08.908820 27894 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:16:08.884576 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42953 (local address 127.0.0.1:41917)
0928 02:16:08.885444 (+ 868us) server_negotiation.cc:206] Beginning negotiation
0928 02:16:08.885456 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0928 02:16:08.885498 (+ 42us) server_negotiation.cc:407] Connection header received
0928 02:16:08.887353 (+ 1855us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:16:08.887378 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:16:08.887865 (+ 487us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:16:08.888242 (+ 377us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:16:08.890243 (+ 2001us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:16:08.891730 (+ 1487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:16:08.893599 (+ 1869us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:16:08.894290 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:16:08.894463 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:16:08.899439 (+ 4976us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:16:08.899459 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:16:08.899482 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:16:08.899537 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:16:08.903625 (+ 4088us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:16:08.904524 (+ 899us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:16:08.904534 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:16:08.904544 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:16:08.904767 (+ 223us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:16:08.905277 (+ 510us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:16:08.905283 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:16:08.905288 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:16:08.906102 (+ 814us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:16:08.906329 (+ 227us) server_negotiation.cc:1035] Waiting for connection context
0928 02:16:08.906932 (+ 603us) server_negotiation.cc:299] Negotiation successful
0928 02:16:08.907382 (+ 450us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":649,"thread_start_us":431,"threads_started":1}
I20260928 02:16:08.909082 27891 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3fc89e191fbc482eacaa67e1b4deba42: No bootstrap required, opened a new log
I20260928 02:16:08.913004 7085 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 02:16:08.917522 27891 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3fc89e191fbc482eacaa67e1b4deba42 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3fc89e191fbc482eacaa67e1b4deba42" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41917 } }
I20260928 02:16:08.918002 27891 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3fc89e191fbc482eacaa67e1b4deba42 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:16:08.918128 27891 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3fc89e191fbc482eacaa67e1b4deba42 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3fc89e191fbc482eacaa67e1b4deba42, State: Initialized, Role: FOLLOWER
I20260928 02:16:08.918828 27891 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3fc89e191fbc482eacaa67e1b4deba42 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3fc89e191fbc482eacaa67e1b4deba42" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41917 } }
I20260928 02:16:08.919086 27891 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3fc89e191fbc482eacaa67e1b4deba42 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 02:16:08.919240 27891 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3fc89e191fbc482eacaa67e1b4deba42 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 02:16:08.919431 27891 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3fc89e191fbc482eacaa67e1b4deba42 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:16:08.921787 27891 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3fc89e191fbc482eacaa67e1b4deba42 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3fc89e191fbc482eacaa67e1b4deba42" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41917 } }
I20260928 02:16:08.922317 27891 leader_election.cc:302] T 00000000000000000000000000000000 P 3fc89e191fbc482eacaa67e1b4deba42 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3fc89e191fbc482eacaa67e1b4deba42; no voters:
I20260928 02:16:08.923022 27891 leader_election.cc:288] T 00000000000000000000000000000000 P 3fc89e191fbc482eacaa67e1b4deba42 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 02:16:08.923123 27901 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3fc89e191fbc482eacaa67e1b4deba42 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:16:08.924497 27901 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3fc89e191fbc482eacaa67e1b4deba42 [term 1 LEADER]: Becoming Leader. State: Replica: 3fc89e191fbc482eacaa67e1b4deba42, State: Running, Role: LEADER
I20260928 02:16:08.925166 27901 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3fc89e191fbc482eacaa67e1b4deba42 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3fc89e191fbc482eacaa67e1b4deba42" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41917 } }
I20260928 02:16:08.925595 27891 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3fc89e191fbc482eacaa67e1b4deba42 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 02:16:08.928671 27902 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3fc89e191fbc482eacaa67e1b4deba42 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3fc89e191fbc482eacaa67e1b4deba42" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3fc89e191fbc482eacaa67e1b4deba42" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41917 } } }
I20260928 02:16:08.928668 27903 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3fc89e191fbc482eacaa67e1b4deba42 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3fc89e191fbc482eacaa67e1b4deba42. Latest consensus state: current_term: 1 leader_uuid: "3fc89e191fbc482eacaa67e1b4deba42" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3fc89e191fbc482eacaa67e1b4deba42" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41917 } } }
I20260928 02:16:08.929149 27902 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3fc89e191fbc482eacaa67e1b4deba42 [sys.catalog]: This master's current role is: LEADER
I20260928 02:16:08.929181 27903 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3fc89e191fbc482eacaa67e1b4deba42 [sys.catalog]: This master's current role is: LEADER
I20260928 02:16:08.934283 27906 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 02:16:08.939534 27906 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 02:16:08.940855 27894 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:16:08.917555 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32967 (local address 127.0.0.1:41917)
0928 02:16:08.917758 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0928 02:16:08.917765 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 02:16:08.918268 (+ 503us) server_negotiation.cc:407] Connection header received
0928 02:16:08.918468 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:16:08.918473 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:16:08.918552 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:16:08.918661 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:16:08.920327 (+ 1666us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:16:08.921149 (+ 822us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:16:08.922472 (+ 1323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:16:08.922960 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:16:08.923127 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:16:08.926326 (+ 3199us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:16:08.926331 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:16:08.926334 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:16:08.926370 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:16:08.935067 (+ 8697us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:16:08.936340 (+ 1273us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:16:08.936349 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:16:08.936355 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:16:08.936621 (+ 266us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:16:08.937258 (+ 637us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:16:08.937266 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:16:08.937271 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:16:08.937861 (+ 590us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:16:08.939791 (+ 1930us) server_negotiation.cc:1035] Waiting for connection context
0928 02:16:08.939981 (+ 190us) server_negotiation.cc:299] Negotiation successful
0928 02:16:08.940293 (+ 312us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260928 02:16:08.950944 27906 catalog_manager.cc:1347] Generated new cluster ID: dbdd7890cfe94b24a40273b77de74afd
I20260928 02:16:08.951083 27906 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 02:16:08.978719 27906 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 02:16:08.980213 27906 catalog_manager.cc:1504] Loading token signing keys...
I20260928 02:16:08.985813 27906 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3fc89e191fbc482eacaa67e1b4deba42: Generated new TSK 0
I20260928 02:16:08.986791 27906 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 02:16:09.002523 7085 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskTlaUuc/build/asan/bin/kudu pid 27824
I20260928 02:16:09.004217 27886 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 02:16:09.098052 7085 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTlaUuc/build/asan/bin/kudu with pid 27824
2026-09-28T02:16:09Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/3 (2116 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/4
Loading random data
Initializing database '/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790561432251112-7085-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 02:16:09 dist-test-slave-86gv krb5kdc[27923](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 02:16:09 dist-test-slave-86gv krb5kdc[27923](info): set up 2 sockets
Sep 28 02:16:09 dist-test-slave-86gv krb5kdc[27923](info): commencing operation
krb5kdc: starting...
W20260928 02:16:10.217734 7085 mini_kdc.cc:121] Time spent starting KDC: real 1.082s user 0.002s sys 0.027s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790561432251112-7085-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790561432251112-7085-0/krb5kdc/test-user.keytab.
Sep 28 02:16:10 dist-test-slave-86gv krb5kdc[27923](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561770, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T02:16:10Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T02:16:10Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790561432251112-7085-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790561432251112-7085-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 02:16:10.496551 7085 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTlaUuc/build/asan/bin/kudu
/tmp/dist-test-taskTlaUuc/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790561432251112-7085-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790561432251112-7085-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790561432251112-7085-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790561432251112-7085-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44061
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790561432251112-7085-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:36323
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:44061
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790561432251112-7085-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790561432251112-7085-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790561432251112-7085-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:16:10.846840 27936 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:16:10.847189 27936 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:16:10.847400 27936 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:16:10.855073 27936 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 02:16:10.855198 27936 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:16:10.855297 27936 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:16:10.855413 27936 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 02:16:10.855477 27936 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 02:16:10.866338 27936 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:36323
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790561432251112-7085-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790561432251112-7085-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:44061
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790561432251112-7085-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44061
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790561432251112-7085-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790561432251112-7085-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 00:51:59 UTC on ccycloud.quasar-axgkwn.root.comops.site
build id 4676464
ASAN enabled
I20260928 02:16:10.867869 27936 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:16:10.869378 27936 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:16:10.882544 27944 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:16:10.882948 27941 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:16:10.882530 27942 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:16:10.883322 27936 server_base.cc:1034] running on GCE node
I20260928 02:16:10.884225 27936 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:16:10.885912 27936 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:16:10.887300 27936 hybrid_clock.cc:648] HybridClock initialized: now 1790561770887220 us; error 58 us; skew 500 ppm
I20260928 02:16:10.887717 27936 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 02:16:10 dist-test-slave-86gv krb5kdc[27923](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561770, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:16:10.891073 27936 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 02:16:10.892764 27936 webserver.cc:460] Webserver started at http://127.0.0.1:35861/ using document root <none> and password file <none>
I20260928 02:16:10.893512 27936 fs_manager.cc:359] Metadata directory not provided
I20260928 02:16:10.893616 27936 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:16:10.893885 27936 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:16:10.897188 27936 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790561432251112-7085-0/minicluster-data/master-0/data/instance:
uuid: "968037d5c22441f8b736861ed0913ed4"
format_stamp: "Formatted at 2026-09-28 02:16:10 on dist-test-slave-86gv"
server_key: "b0a4564c46a19816ed31ae6ed5c77a5b"
server_key_iv: "acf9f97921124c6357a3643d65972b5a"
server_key_version: "encryptionkey@0"
I20260928 02:16:10.898001 27936 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790561432251112-7085-0/minicluster-data/master-0/wal/instance:
uuid: "968037d5c22441f8b736861ed0913ed4"
format_stamp: "Formatted at 2026-09-28 02:16:10 on dist-test-slave-86gv"
server_key: "b0a4564c46a19816ed31ae6ed5c77a5b"
server_key_iv: "acf9f97921124c6357a3643d65972b5a"
server_key_version: "encryptionkey@0"
I20260928 02:16:10.903744 27936 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260928 02:16:10.907714 27951 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:16:10.909274 27936 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260928 02:16:10.909440 27936 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790561432251112-7085-0/minicluster-data/master-0/data,/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790561432251112-7085-0/minicluster-data/master-0/wal
uuid: "968037d5c22441f8b736861ed0913ed4"
format_stamp: "Formatted at 2026-09-28 02:16:10 on dist-test-slave-86gv"
server_key: "b0a4564c46a19816ed31ae6ed5c77a5b"
server_key_iv: "acf9f97921124c6357a3643d65972b5a"
server_key_version: "encryptionkey@0"
I20260928 02:16:10.909629 27936 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790561432251112-7085-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790561432251112-7085-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790561432251112-7085-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:16:10.923472 27936 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:16:10.928411 27936 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:16:10.928740 27936 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:16:10.947817 27936 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:44061
I20260928 02:16:10.947938 28002 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:44061 every 8 connection(s)
I20260928 02:16:10.949360 27936 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790561432251112-7085-0/minicluster-data/master-0/data/info.pb
I20260928 02:16:10.953473 28003 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:16:10.953800 7085 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTlaUuc/build/asan/bin/kudu as pid 27936
I20260928 02:16:10.953996 7085 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790561432251112-7085-0/minicluster-data/master-0/wal/instance
I20260928 02:16:10.954520 7085 external_mini_cluster.cc:1428] Setting key 9a8e7c666c8bb23cc71b8444ffed5071
I20260928 02:16:10.970041 28003 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 968037d5c22441f8b736861ed0913ed4: Bootstrap starting.
I20260928 02:16:10.970098 28006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:16:10.956460 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55609 (local address 127.0.0.1:44061)
0928 02:16:10.957372 (+ 912us) server_negotiation.cc:206] Beginning negotiation
0928 02:16:10.957394 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0928 02:16:10.957446 (+ 52us) server_negotiation.cc:407] Connection header received
0928 02:16:10.959297 (+ 1851us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:16:10.959331 (+ 34us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:16:10.959879 (+ 548us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:16:10.960378 (+ 499us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:16:10.960966 (+ 588us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:16:10.963001 (+ 2035us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:16:10.964480 (+ 1479us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:16:10.965592 (+ 1112us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:16:10.965809 (+ 217us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:16:10.967021 (+ 1212us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:16:10.967032 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:16:10.967059 (+ 27us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 02:16:10.967832 (+ 773us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:16:10.967956 (+ 124us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 02:16:10.967991 (+ 35us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:16:10.968095 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0928 02:16:10.968389 (+ 294us) server_negotiation.cc:299] Negotiation successful
0928 02:16:10.968690 (+ 301us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":653,"thread_start_us":428,"threads_started":1}
I20260928 02:16:10.974267 28003 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 968037d5c22441f8b736861ed0913ed4: Neither blocks nor log segments found. Creating new log.
I20260928 02:16:10.974860 7085 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 02:16:10.975556 28003 log.cc:824] T 00000000000000000000000000000000 P 968037d5c22441f8b736861ed0913ed4: Log is configured to *not* fsync() on all Append() calls
I20260928 02:16:10.978660 28003 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 968037d5c22441f8b736861ed0913ed4: No bootstrap required, opened a new log
I20260928 02:16:10.987109 28006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:16:10.979615 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43191 (local address 127.0.0.1:44061)
0928 02:16:10.979807 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0928 02:16:10.979814 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 02:16:10.980269 (+ 455us) server_negotiation.cc:407] Connection header received
0928 02:16:10.980437 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:16:10.980443 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:16:10.980540 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:16:10.980656 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:16:10.981235 (+ 579us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:16:10.982398 (+ 1163us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:16:10.983884 (+ 1486us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:16:10.984627 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:16:10.984773 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:16:10.986200 (+ 1427us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:16:10.986206 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:16:10.986211 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 02:16:10.986234 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:16:10.986291 (+ 57us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 02:16:10.986313 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:16:10.986429 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0928 02:16:10.986615 (+ 186us) server_negotiation.cc:299] Negotiation successful
0928 02:16:10.986728 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260928 02:16:10.989225 28003 raft_consensus.cc:348] T 00000000000000000000000000000000 P 968037d5c22441f8b736861ed0913ed4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "968037d5c22441f8b736861ed0913ed4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44061 } }
I20260928 02:16:10.989651 28003 raft_consensus.cc:374] T 00000000000000000000000000000000 P 968037d5c22441f8b736861ed0913ed4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:16:10.989779 28003 raft_consensus.cc:729] T 00000000000000000000000000000000 P 968037d5c22441f8b736861ed0913ed4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 968037d5c22441f8b736861ed0913ed4, State: Initialized, Role: FOLLOWER
I20260928 02:16:10.990423 28003 consensus_queue.cc:260] T 00000000000000000000000000000000 P 968037d5c22441f8b736861ed0913ed4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "968037d5c22441f8b736861ed0913ed4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44061 } }
I20260928 02:16:10.990688 28003 raft_consensus.cc:388] T 00000000000000000000000000000000 P 968037d5c22441f8b736861ed0913ed4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 02:16:10.990854 28003 raft_consensus.cc:482] T 00000000000000000000000000000000 P 968037d5c22441f8b736861ed0913ed4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 02:16:10.991034 28003 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 968037d5c22441f8b736861ed0913ed4 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:16:10.993503 28003 raft_consensus.cc:504] T 00000000000000000000000000000000 P 968037d5c22441f8b736861ed0913ed4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "968037d5c22441f8b736861ed0913ed4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44061 } }
I20260928 02:16:10.994131 28003 leader_election.cc:302] T 00000000000000000000000000000000 P 968037d5c22441f8b736861ed0913ed4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 968037d5c22441f8b736861ed0913ed4; no voters:
I20260928 02:16:10.994804 28003 leader_election.cc:288] T 00000000000000000000000000000000 P 968037d5c22441f8b736861ed0913ed4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 02:16:10.994942 28013 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 968037d5c22441f8b736861ed0913ed4 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:16:10.995675 28013 raft_consensus.cc:686] T 00000000000000000000000000000000 P 968037d5c22441f8b736861ed0913ed4 [term 1 LEADER]: Becoming Leader. State: Replica: 968037d5c22441f8b736861ed0913ed4, State: Running, Role: LEADER
I20260928 02:16:10.996316 28013 consensus_queue.cc:237] T 00000000000000000000000000000000 P 968037d5c22441f8b736861ed0913ed4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "968037d5c22441f8b736861ed0913ed4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44061 } }
I20260928 02:16:10.997170 28003 sys_catalog.cc:564] T 00000000000000000000000000000000 P 968037d5c22441f8b736861ed0913ed4 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 02:16:10.999333 28014 sys_catalog.cc:455] T 00000000000000000000000000000000 P 968037d5c22441f8b736861ed0913ed4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "968037d5c22441f8b736861ed0913ed4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "968037d5c22441f8b736861ed0913ed4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44061 } } }
I20260928 02:16:10.999713 28014 sys_catalog.cc:458] T 00000000000000000000000000000000 P 968037d5c22441f8b736861ed0913ed4 [sys.catalog]: This master's current role is: LEADER
I20260928 02:16:11.000136 28015 sys_catalog.cc:455] T 00000000000000000000000000000000 P 968037d5c22441f8b736861ed0913ed4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 968037d5c22441f8b736861ed0913ed4. Latest consensus state: current_term: 1 leader_uuid: "968037d5c22441f8b736861ed0913ed4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "968037d5c22441f8b736861ed0913ed4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44061 } } }
I20260928 02:16:11.000437 28015 sys_catalog.cc:458] T 00000000000000000000000000000000 P 968037d5c22441f8b736861ed0913ed4 [sys.catalog]: This master's current role is: LEADER
I20260928 02:16:11.001933 28019 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 02:16:11.007938 28019 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 02:16:11.018533 28019 catalog_manager.cc:1347] Generated new cluster ID: 46b7dc44b4ce475fad3e59f6f53e4a6f
I20260928 02:16:11.018685 28019 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 02:16:11.033926 28019 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 02:16:11.035987 28019 catalog_manager.cc:1504] Loading token signing keys...
I20260928 02:16:11.047344 28019 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 968037d5c22441f8b736861ed0913ed4: Generated new TSK 0
I20260928 02:16:11.048620 28019 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 02:16:11.056090 7085 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskTlaUuc/build/asan/bin/kudu pid 27936
W20260928 02:16:11.057251 27998 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:55609
W20260928 02:16:11.058068 7085 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskTlaUuc/build/asan/bin/kudu pid 27936: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260928 02:16:11.058279 7085 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTlaUuc/build/asan/bin/kudu with pid 27936
2026-09-28T02:16:11Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/4 (1966 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/5
Loading random data
Initializing database '/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790561432251112-7085-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 02:16:11 dist-test-slave-86gv krb5kdc[28034](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 02:16:11 dist-test-slave-86gv krb5kdc[28034](info): set up 2 sockets
Sep 28 02:16:11 dist-test-slave-86gv krb5kdc[28034](info): commencing operation
krb5kdc: starting...
W20260928 02:16:12.198307 7085 mini_kdc.cc:121] Time spent starting KDC: real 1.094s user 0.004s sys 0.029s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790561432251112-7085-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790561432251112-7085-0/krb5kdc/test-user.keytab.
Sep 28 02:16:12 dist-test-slave-86gv krb5kdc[28034](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561772, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T02:16:12Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T02:16:12Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790561432251112-7085-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790561432251112-7085-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 02:16:12.481423 7085 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTlaUuc/build/asan/bin/kudu
/tmp/dist-test-taskTlaUuc/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790561432251112-7085-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790561432251112-7085-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790561432251112-7085-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790561432251112-7085-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40695
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790561432251112-7085-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:33517
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:40695
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790561432251112-7085-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790561432251112-7085-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790561432251112-7085-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:16:12.845248 28047 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:16:12.845605 28047 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:16:12.845707 28047 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:16:12.853477 28047 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 02:16:12.853605 28047 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:16:12.853653 28047 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:16:12.853714 28047 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 02:16:12.853760 28047 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 02:16:12.864180 28047 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:33517
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790561432251112-7085-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790561432251112-7085-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:40695
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790561432251112-7085-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40695
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790561432251112-7085-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790561432251112-7085-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 00:51:59 UTC on ccycloud.quasar-axgkwn.root.comops.site
build id 4676464
ASAN enabled
I20260928 02:16:12.865573 28047 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:16:12.866959 28047 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:16:12.879545 28052 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:16:12.879648 28055 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:16:12.879622 28053 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:16:12.880421 28047 server_base.cc:1034] running on GCE node
I20260928 02:16:12.881313 28047 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:16:12.882906 28047 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:16:12.884135 28047 hybrid_clock.cc:648] HybridClock initialized: now 1790561772884086 us; error 39 us; skew 500 ppm
I20260928 02:16:12.884523 28047 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 02:16:12 dist-test-slave-86gv krb5kdc[28034](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561772, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:16:12.901491 28047 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 02:16:12.903291 28047 webserver.cc:460] Webserver started at http://127.0.0.1:46235/ using document root <none> and password file <none>
I20260928 02:16:12.904078 28047 fs_manager.cc:359] Metadata directory not provided
I20260928 02:16:12.904214 28047 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:16:12.904565 28047 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:16:12.907788 28047 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790561432251112-7085-0/minicluster-data/master-0/data/instance:
uuid: "f22a2c01cd8d4c5c9ed4c3f5c7319b27"
format_stamp: "Formatted at 2026-09-28 02:16:12 on dist-test-slave-86gv"
server_key: "f756893783eaa6c3c6a3566fd55e44fe"
server_key_iv: "e4793134ba723ad54027f2badb45d0e1"
server_key_version: "encryptionkey@0"
I20260928 02:16:12.908649 28047 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790561432251112-7085-0/minicluster-data/master-0/wal/instance:
uuid: "f22a2c01cd8d4c5c9ed4c3f5c7319b27"
format_stamp: "Formatted at 2026-09-28 02:16:12 on dist-test-slave-86gv"
server_key: "f756893783eaa6c3c6a3566fd55e44fe"
server_key_iv: "e4793134ba723ad54027f2badb45d0e1"
server_key_version: "encryptionkey@0"
I20260928 02:16:12.914201 28047 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260928 02:16:12.917879 28062 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:16:12.919381 28047 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260928 02:16:12.919584 28047 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790561432251112-7085-0/minicluster-data/master-0/data,/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790561432251112-7085-0/minicluster-data/master-0/wal
uuid: "f22a2c01cd8d4c5c9ed4c3f5c7319b27"
format_stamp: "Formatted at 2026-09-28 02:16:12 on dist-test-slave-86gv"
server_key: "f756893783eaa6c3c6a3566fd55e44fe"
server_key_iv: "e4793134ba723ad54027f2badb45d0e1"
server_key_version: "encryptionkey@0"
I20260928 02:16:12.919808 28047 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790561432251112-7085-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790561432251112-7085-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790561432251112-7085-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:16:12.926077 28047 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:16:12.930639 28047 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:16:12.930953 28047 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:16:12.950726 28047 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:40695
I20260928 02:16:12.950819 28113 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:40695 every 8 connection(s)
I20260928 02:16:12.952243 28047 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790561432251112-7085-0/minicluster-data/master-0/data/info.pb
I20260928 02:16:12.956218 28114 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:16:12.960026 7085 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTlaUuc/build/asan/bin/kudu as pid 28047
I20260928 02:16:12.960237 7085 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790561432251112-7085-0/minicluster-data/master-0/wal/instance
I20260928 02:16:12.960744 7085 external_mini_cluster.cc:1428] Setting key dd7ca31da9c08ce9ec897c45ff746ed4
I20260928 02:16:12.968971 28117 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:16:12.962707 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53981 (local address 127.0.0.1:40695)
0928 02:16:12.963584 (+ 877us) server_negotiation.cc:206] Beginning negotiation
0928 02:16:12.963596 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0928 02:16:12.963637 (+ 41us) server_negotiation.cc:407] Connection header received
0928 02:16:12.965371 (+ 1734us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:16:12.965401 (+ 30us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:16:12.965796 (+ 395us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:16:12.966145 (+ 349us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:16:12.967279 (+ 1134us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:16:12.967285 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:16:12.967297 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 02:16:12.967312 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:16:12.967370 (+ 58us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 02:16:12.967390 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:16:12.967602 (+ 212us) server_negotiation.cc:1035] Waiting for connection context
0928 02:16:12.967727 (+ 125us) server_negotiation.cc:299] Negotiation successful
0928 02:16:12.967942 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":640,"thread_start_us":409,"threads_started":1}
I20260928 02:16:12.969635 28114 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f22a2c01cd8d4c5c9ed4c3f5c7319b27: Bootstrap starting.
I20260928 02:16:12.972949 7085 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 02:16:12.973100 28114 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f22a2c01cd8d4c5c9ed4c3f5c7319b27: Neither blocks nor log segments found. Creating new log.
I20260928 02:16:12.974169 28114 log.cc:824] T 00000000000000000000000000000000 P f22a2c01cd8d4c5c9ed4c3f5c7319b27: Log is configured to *not* fsync() on all Append() calls
I20260928 02:16:12.977350 28114 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f22a2c01cd8d4c5c9ed4c3f5c7319b27: No bootstrap required, opened a new log
I20260928 02:16:12.980137 28117 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:16:12.977554 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38541 (local address 127.0.0.1:40695)
0928 02:16:12.977734 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0928 02:16:12.977740 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 02:16:12.978229 (+ 489us) server_negotiation.cc:407] Connection header received
0928 02:16:12.978367 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:16:12.978372 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:16:12.978441 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:16:12.978554 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:16:12.979463 (+ 909us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:16:12.979470 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:16:12.979475 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 02:16:12.979498 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:16:12.979550 (+ 52us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 02:16:12.979570 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:16:12.979713 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0928 02:16:12.979767 (+ 54us) server_negotiation.cc:299] Negotiation successful
0928 02:16:12.979826 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260928 02:16:12.986248 28114 raft_consensus.cc:348] T 00000000000000000000000000000000 P f22a2c01cd8d4c5c9ed4c3f5c7319b27 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f22a2c01cd8d4c5c9ed4c3f5c7319b27" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40695 } }
I20260928 02:16:12.986668 28114 raft_consensus.cc:374] T 00000000000000000000000000000000 P f22a2c01cd8d4c5c9ed4c3f5c7319b27 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:16:12.986789 28114 raft_consensus.cc:729] T 00000000000000000000000000000000 P f22a2c01cd8d4c5c9ed4c3f5c7319b27 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f22a2c01cd8d4c5c9ed4c3f5c7319b27, State: Initialized, Role: FOLLOWER
I20260928 02:16:12.987434 28114 consensus_queue.cc:260] T 00000000000000000000000000000000 P f22a2c01cd8d4c5c9ed4c3f5c7319b27 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f22a2c01cd8d4c5c9ed4c3f5c7319b27" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40695 } }
I20260928 02:16:12.987684 28114 raft_consensus.cc:388] T 00000000000000000000000000000000 P f22a2c01cd8d4c5c9ed4c3f5c7319b27 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 02:16:12.987844 28114 raft_consensus.cc:482] T 00000000000000000000000000000000 P f22a2c01cd8d4c5c9ed4c3f5c7319b27 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 02:16:12.988044 28114 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f22a2c01cd8d4c5c9ed4c3f5c7319b27 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:16:12.990900 28114 raft_consensus.cc:504] T 00000000000000000000000000000000 P f22a2c01cd8d4c5c9ed4c3f5c7319b27 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f22a2c01cd8d4c5c9ed4c3f5c7319b27" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40695 } }
I20260928 02:16:12.991448 28114 leader_election.cc:302] T 00000000000000000000000000000000 P f22a2c01cd8d4c5c9ed4c3f5c7319b27 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f22a2c01cd8d4c5c9ed4c3f5c7319b27; no voters:
I20260928 02:16:12.992093 28114 leader_election.cc:288] T 00000000000000000000000000000000 P f22a2c01cd8d4c5c9ed4c3f5c7319b27 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 02:16:12.992268 28124 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f22a2c01cd8d4c5c9ed4c3f5c7319b27 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:16:12.992992 28124 raft_consensus.cc:686] T 00000000000000000000000000000000 P f22a2c01cd8d4c5c9ed4c3f5c7319b27 [term 1 LEADER]: Becoming Leader. State: Replica: f22a2c01cd8d4c5c9ed4c3f5c7319b27, State: Running, Role: LEADER
I20260928 02:16:12.993616 28124 consensus_queue.cc:237] T 00000000000000000000000000000000 P f22a2c01cd8d4c5c9ed4c3f5c7319b27 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f22a2c01cd8d4c5c9ed4c3f5c7319b27" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40695 } }
I20260928 02:16:12.994413 28114 sys_catalog.cc:564] T 00000000000000000000000000000000 P f22a2c01cd8d4c5c9ed4c3f5c7319b27 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 02:16:12.996378 28125 sys_catalog.cc:455] T 00000000000000000000000000000000 P f22a2c01cd8d4c5c9ed4c3f5c7319b27 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f22a2c01cd8d4c5c9ed4c3f5c7319b27" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f22a2c01cd8d4c5c9ed4c3f5c7319b27" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40695 } } }
I20260928 02:16:12.996920 28125 sys_catalog.cc:458] T 00000000000000000000000000000000 P f22a2c01cd8d4c5c9ed4c3f5c7319b27 [sys.catalog]: This master's current role is: LEADER
I20260928 02:16:12.997867 28129 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 02:16:12.997659 28126 sys_catalog.cc:455] T 00000000000000000000000000000000 P f22a2c01cd8d4c5c9ed4c3f5c7319b27 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f22a2c01cd8d4c5c9ed4c3f5c7319b27. Latest consensus state: current_term: 1 leader_uuid: "f22a2c01cd8d4c5c9ed4c3f5c7319b27" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f22a2c01cd8d4c5c9ed4c3f5c7319b27" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40695 } } }
I20260928 02:16:12.998019 28126 sys_catalog.cc:458] T 00000000000000000000000000000000 P f22a2c01cd8d4c5c9ed4c3f5c7319b27 [sys.catalog]: This master's current role is: LEADER
I20260928 02:16:13.003818 28129 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 02:16:13.012785 28129 catalog_manager.cc:1347] Generated new cluster ID: 8e208031d15149fc8cd644ae0fd6c485
I20260928 02:16:13.013000 28129 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 02:16:13.030117 28129 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 02:16:13.030704 28129 catalog_manager.cc:1504] Loading token signing keys...
I20260928 02:16:13.044154 28129 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f22a2c01cd8d4c5c9ed4c3f5c7319b27: Generated new TSK 0
I20260928 02:16:13.045351 28129 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 02:16:13.061121 7085 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskTlaUuc/build/asan/bin/kudu pid 28047
W20260928 02:16:13.062297 28109 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:53981
W20260928 02:16:13.063081 7085 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskTlaUuc/build/asan/bin/kudu pid 28047: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260928 02:16:13.063246 7085 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTlaUuc/build/asan/bin/kudu with pid 28047
2026-09-28T02:16:13Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/5 (1997 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/6
Loading random data
Initializing database '/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790561432251112-7085-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 02:16:13 dist-test-slave-86gv krb5kdc[28145](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 02:16:13 dist-test-slave-86gv krb5kdc[28145](info): set up 2 sockets
Sep 28 02:16:13 dist-test-slave-86gv krb5kdc[28145](info): commencing operation
krb5kdc: starting...
W20260928 02:16:14.177886 7085 mini_kdc.cc:121] Time spent starting KDC: real 1.076s user 0.002s sys 0.028s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790561432251112-7085-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790561432251112-7085-0/krb5kdc/test-user.keytab.
Sep 28 02:16:14 dist-test-slave-86gv krb5kdc[28145](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561774, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T02:16:14Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T02:16:14Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790561432251112-7085-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790561432251112-7085-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 02:16:14.461388 7085 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTlaUuc/build/asan/bin/kudu
/tmp/dist-test-taskTlaUuc/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790561432251112-7085-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790561432251112-7085-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790561432251112-7085-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790561432251112-7085-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44639
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790561432251112-7085-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:40569
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:44639
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790561432251112-7085-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790561432251112-7085-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790561432251112-7085-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:16:14.820627 28158 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:16:14.821049 28158 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:16:14.821159 28158 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:16:14.828810 28158 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 02:16:14.829245 28158 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:16:14.829348 28158 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:16:14.829411 28158 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 02:16:14.829460 28158 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 02:16:14.840024 28158 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:40569
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790561432251112-7085-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790561432251112-7085-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:44639
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790561432251112-7085-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44639
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790561432251112-7085-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790561432251112-7085-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 00:51:59 UTC on ccycloud.quasar-axgkwn.root.comops.site
build id 4676464
ASAN enabled
I20260928 02:16:14.841595 28158 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:16:14.842934 28158 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:16:14.856272 28163 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:16:14.856240 28164 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:16:14.856200 28166 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:16:14.856827 28158 server_base.cc:1034] running on GCE node
I20260928 02:16:14.857851 28158 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:16:14.859535 28158 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:16:14.860754 28158 hybrid_clock.cc:648] HybridClock initialized: now 1790561774860687 us; error 58 us; skew 500 ppm
I20260928 02:16:14.861168 28158 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 02:16:14 dist-test-slave-86gv krb5kdc[28145](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561774, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:16:14.864658 28158 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 02:16:14.866323 28158 webserver.cc:460] Webserver started at http://127.0.0.1:43389/ using document root <none> and password file <none>
I20260928 02:16:14.867134 28158 fs_manager.cc:359] Metadata directory not provided
I20260928 02:16:14.867295 28158 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:16:14.867611 28158 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:16:14.870834 28158 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790561432251112-7085-0/minicluster-data/master-0/data/instance:
uuid: "1c15705241724665bc596706b8f48d3d"
format_stamp: "Formatted at 2026-09-28 02:16:14 on dist-test-slave-86gv"
server_key: "9b81a03a892e2f8936f4ad6ccf088b13"
server_key_iv: "51c35cbabef5d294b5d2c3e292617ea0"
server_key_version: "encryptionkey@0"
I20260928 02:16:14.871645 28158 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790561432251112-7085-0/minicluster-data/master-0/wal/instance:
uuid: "1c15705241724665bc596706b8f48d3d"
format_stamp: "Formatted at 2026-09-28 02:16:14 on dist-test-slave-86gv"
server_key: "9b81a03a892e2f8936f4ad6ccf088b13"
server_key_iv: "51c35cbabef5d294b5d2c3e292617ea0"
server_key_version: "encryptionkey@0"
I20260928 02:16:14.877380 28158 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.003s
I20260928 02:16:14.881467 28173 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:16:14.882906 28158 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260928 02:16:14.883116 28158 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790561432251112-7085-0/minicluster-data/master-0/data,/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790561432251112-7085-0/minicluster-data/master-0/wal
uuid: "1c15705241724665bc596706b8f48d3d"
format_stamp: "Formatted at 2026-09-28 02:16:14 on dist-test-slave-86gv"
server_key: "9b81a03a892e2f8936f4ad6ccf088b13"
server_key_iv: "51c35cbabef5d294b5d2c3e292617ea0"
server_key_version: "encryptionkey@0"
I20260928 02:16:14.883332 28158 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790561432251112-7085-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790561432251112-7085-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790561432251112-7085-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:16:14.897456 28158 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:16:14.902444 28158 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:16:14.902745 28158 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:16:14.922174 28158 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:44639
I20260928 02:16:14.922194 28224 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:44639 every 8 connection(s)
I20260928 02:16:14.923703 28158 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790561432251112-7085-0/minicluster-data/master-0/data/info.pb
I20260928 02:16:14.927747 28225 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:16:14.930334 7085 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTlaUuc/build/asan/bin/kudu as pid 28158
I20260928 02:16:14.930559 7085 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790561432251112-7085-0/minicluster-data/master-0/wal/instance
I20260928 02:16:14.931015 7085 external_mini_cluster.cc:1428] Setting key b1ab8a10a30405a31cde8746e522a139
I20260928 02:16:14.939899 28225 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1c15705241724665bc596706b8f48d3d: Bootstrap starting.
I20260928 02:16:14.943149 28225 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1c15705241724665bc596706b8f48d3d: Neither blocks nor log segments found. Creating new log.
I20260928 02:16:14.944294 28225 log.cc:824] T 00000000000000000000000000000000 P 1c15705241724665bc596706b8f48d3d: Log is configured to *not* fsync() on all Append() calls
Sep 28 02:16:14 dist-test-slave-86gv krb5kdc[28145](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561774, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260928 02:16:14.947683 28225 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1c15705241724665bc596706b8f48d3d: No bootstrap required, opened a new log
I20260928 02:16:14.954855 28228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:16:14.933037 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.49:45103 (local address 127.0.0.1:44639)
0928 02:16:14.933912 (+ 875us) server_negotiation.cc:206] Beginning negotiation
0928 02:16:14.933924 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0928 02:16:14.933978 (+ 54us) server_negotiation.cc:407] Connection header received
0928 02:16:14.935700 (+ 1722us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:16:14.935726 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:16:14.936248 (+ 522us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:16:14.936692 (+ 444us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:16:14.938244 (+ 1552us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:16:14.939550 (+ 1306us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:16:14.940950 (+ 1400us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:16:14.941645 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:16:14.941859 (+ 214us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:16:14.946430 (+ 4571us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:16:14.946439 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:16:14.946457 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:16:14.946494 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:16:14.950021 (+ 3527us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:16:14.950986 (+ 965us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:16:14.950997 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:16:14.951004 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:16:14.951188 (+ 184us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:16:14.951688 (+ 500us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:16:14.951698 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:16:14.951700 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:16:14.952400 (+ 700us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:16:14.952752 (+ 352us) server_negotiation.cc:1035] Waiting for connection context
0928 02:16:14.953320 (+ 568us) server_negotiation.cc:299] Negotiation successful
0928 02:16:14.953691 (+ 371us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":641,"thread_start_us":383,"threads_started":1}
I20260928 02:16:14.956183 28225 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1c15705241724665bc596706b8f48d3d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1c15705241724665bc596706b8f48d3d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44639 } }
I20260928 02:16:14.956588 28225 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1c15705241724665bc596706b8f48d3d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:16:14.956717 28225 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1c15705241724665bc596706b8f48d3d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c15705241724665bc596706b8f48d3d, State: Initialized, Role: FOLLOWER
I20260928 02:16:14.957437 28225 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1c15705241724665bc596706b8f48d3d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1c15705241724665bc596706b8f48d3d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44639 } }
I20260928 02:16:14.957728 28225 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1c15705241724665bc596706b8f48d3d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 02:16:14.957880 28225 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1c15705241724665bc596706b8f48d3d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 02:16:14.958068 28225 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1c15705241724665bc596706b8f48d3d [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:16:14.959525 7085 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 02:16:14.961524 28225 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1c15705241724665bc596706b8f48d3d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1c15705241724665bc596706b8f48d3d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44639 } }
I20260928 02:16:14.962214 28225 leader_election.cc:302] T 00000000000000000000000000000000 P 1c15705241724665bc596706b8f48d3d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1c15705241724665bc596706b8f48d3d; no voters:
I20260928 02:16:14.962994 28225 leader_election.cc:288] T 00000000000000000000000000000000 P 1c15705241724665bc596706b8f48d3d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 02:16:14.963194 28230 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1c15705241724665bc596706b8f48d3d [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:16:14.965193 28230 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1c15705241724665bc596706b8f48d3d [term 1 LEADER]: Becoming Leader. State: Replica: 1c15705241724665bc596706b8f48d3d, State: Running, Role: LEADER
I20260928 02:16:14.965938 28225 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1c15705241724665bc596706b8f48d3d [sys.catalog]: configured and running, proceeding with master startup.
I20260928 02:16:14.965889 28230 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1c15705241724665bc596706b8f48d3d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1c15705241724665bc596706b8f48d3d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44639 } }
I20260928 02:16:14.970626 28234 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1c15705241724665bc596706b8f48d3d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1c15705241724665bc596706b8f48d3d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1c15705241724665bc596706b8f48d3d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44639 } } }
I20260928 02:16:14.971134 28234 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1c15705241724665bc596706b8f48d3d [sys.catalog]: This master's current role is: LEADER
I20260928 02:16:14.975615 28235 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1c15705241724665bc596706b8f48d3d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1c15705241724665bc596706b8f48d3d. Latest consensus state: current_term: 1 leader_uuid: "1c15705241724665bc596706b8f48d3d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1c15705241724665bc596706b8f48d3d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44639 } } }
I20260928 02:16:14.975982 28235 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1c15705241724665bc596706b8f48d3d [sys.catalog]: This master's current role is: LEADER
I20260928 02:16:14.977669 28245 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 02:16:14.982937 28245 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 02:16:14.991930 28228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:16:14.966227 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.49:35073 (local address 127.0.0.1:44639)
0928 02:16:14.966401 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0928 02:16:14.966408 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 02:16:14.972491 (+ 6083us) server_negotiation.cc:407] Connection header received
0928 02:16:14.972746 (+ 255us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:16:14.972755 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:16:14.972871 (+ 116us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:16:14.973056 (+ 185us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:16:14.975565 (+ 2509us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:16:14.977281 (+ 1716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:16:14.979453 (+ 2172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:16:14.980207 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:16:14.980362 (+ 155us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:16:14.983840 (+ 3478us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:16:14.983847 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:16:14.983855 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:16:14.983911 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:16:14.988112 (+ 4201us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:16:14.989272 (+ 1160us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:16:14.989279 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:16:14.989283 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:16:14.989474 (+ 191us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:16:14.990051 (+ 577us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:16:14.990056 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:16:14.990060 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:16:14.990538 (+ 478us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:16:14.990836 (+ 298us) server_negotiation.cc:1035] Waiting for connection context
0928 02:16:14.991245 (+ 409us) server_negotiation.cc:299] Negotiation successful
0928 02:16:14.991496 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260928 02:16:14.994987 28245 catalog_manager.cc:1347] Generated new cluster ID: b213e07eeb8f4db0ab1577a3c04a3878
I20260928 02:16:14.995111 28245 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 02:16:15.030027 28245 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 02:16:15.031502 28245 catalog_manager.cc:1504] Loading token signing keys...
I20260928 02:16:15.042411 28245 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1c15705241724665bc596706b8f48d3d: Generated new TSK 0
I20260928 02:16:15.043401 28245 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 02:16:15.053574 7085 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskTlaUuc/build/asan/bin/kudu pid 28158
I20260928 02:16:15.055163 28220 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 02:16:15.151081 7085 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTlaUuc/build/asan/bin/kudu with pid 28158
W20260928 02:16:15.169938 28143 connection.cc:535] client connection to 127.0.0.1:44639 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:44639): Connection reset by peer (error 104)
2026-09-28T02:16:15Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/6 (2088 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/7
Loading random data
Initializing database '/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790561432251112-7085-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 02:16:15 dist-test-slave-86gv krb5kdc[28257](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 02:16:15 dist-test-slave-86gv krb5kdc[28257](info): set up 2 sockets
Sep 28 02:16:15 dist-test-slave-86gv krb5kdc[28257](info): commencing operation
krb5kdc: starting...
W20260928 02:16:16.270098 7085 mini_kdc.cc:121] Time spent starting KDC: real 1.080s user 0.004s sys 0.025s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790561432251112-7085-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790561432251112-7085-0/krb5kdc/test-user.keytab.
Sep 28 02:16:16 dist-test-slave-86gv krb5kdc[28257](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561776, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T02:16:16Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T02:16:16Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790561432251112-7085-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790561432251112-7085-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 02:16:16.553855 7085 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTlaUuc/build/asan/bin/kudu
/tmp/dist-test-taskTlaUuc/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790561432251112-7085-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790561432251112-7085-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790561432251112-7085-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790561432251112-7085-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46493
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790561432251112-7085-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:44573
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:46493
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790561432251112-7085-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790561432251112-7085-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790561432251112-7085-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:16:16.910830 28270 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:16:16.911142 28270 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:16:16.911312 28270 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:16:16.918727 28270 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 02:16:16.918864 28270 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:16:16.918962 28270 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:16:16.919071 28270 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 02:16:16.919144 28270 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 02:16:16.929603 28270 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:44573
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790561432251112-7085-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790561432251112-7085-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:46493
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790561432251112-7085-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46493
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790561432251112-7085-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790561432251112-7085-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 00:51:59 UTC on ccycloud.quasar-axgkwn.root.comops.site
build id 4676464
ASAN enabled
I20260928 02:16:16.931068 28270 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:16:16.932452 28270 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:16:16.946465 28278 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:16:16.946530 28276 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:16:16.946497 28275 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:16:16.948025 28270 server_base.cc:1034] running on GCE node
I20260928 02:16:16.948998 28270 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:16:16.950670 28270 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:16:16.951942 28270 hybrid_clock.cc:648] HybridClock initialized: now 1790561776951876 us; error 64 us; skew 500 ppm
I20260928 02:16:16.952355 28270 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 02:16:16 dist-test-slave-86gv krb5kdc[28257](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561776, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:16:16.955971 28270 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 02:16:16.958097 28270 webserver.cc:460] Webserver started at http://127.0.0.1:38583/ using document root <none> and password file <none>
I20260928 02:16:16.958928 28270 fs_manager.cc:359] Metadata directory not provided
I20260928 02:16:16.959092 28270 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:16:16.959467 28270 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:16:16.963004 28270 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790561432251112-7085-0/minicluster-data/master-0/data/instance:
uuid: "f0d89bccc01e45b8b4362012f52a5d7b"
format_stamp: "Formatted at 2026-09-28 02:16:16 on dist-test-slave-86gv"
server_key: "bdc782d273057ed0315eff9df230c75a"
server_key_iv: "b7d43b8e1a687cc98de598ce27a77428"
server_key_version: "encryptionkey@0"
I20260928 02:16:16.963910 28270 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790561432251112-7085-0/minicluster-data/master-0/wal/instance:
uuid: "f0d89bccc01e45b8b4362012f52a5d7b"
format_stamp: "Formatted at 2026-09-28 02:16:16 on dist-test-slave-86gv"
server_key: "bdc782d273057ed0315eff9df230c75a"
server_key_iv: "b7d43b8e1a687cc98de598ce27a77428"
server_key_version: "encryptionkey@0"
I20260928 02:16:16.970050 28270 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.008s sys 0.000s
I20260928 02:16:16.973953 28285 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:16:16.975535 28270 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260928 02:16:16.975749 28270 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790561432251112-7085-0/minicluster-data/master-0/data,/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790561432251112-7085-0/minicluster-data/master-0/wal
uuid: "f0d89bccc01e45b8b4362012f52a5d7b"
format_stamp: "Formatted at 2026-09-28 02:16:16 on dist-test-slave-86gv"
server_key: "bdc782d273057ed0315eff9df230c75a"
server_key_iv: "b7d43b8e1a687cc98de598ce27a77428"
server_key_version: "encryptionkey@0"
I20260928 02:16:16.975994 28270 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790561432251112-7085-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790561432251112-7085-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790561432251112-7085-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:16:17.005375 28270 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:16:17.009999 28270 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:16:17.010308 28270 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:16:17.029716 28270 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:46493
I20260928 02:16:17.029739 28336 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:46493 every 8 connection(s)
I20260928 02:16:17.031275 28270 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790561432251112-7085-0/minicluster-data/master-0/data/info.pb
I20260928 02:16:17.032884 7085 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTlaUuc/build/asan/bin/kudu as pid 28270
I20260928 02:16:17.033090 7085 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790561432251112-7085-0/minicluster-data/master-0/wal/instance
I20260928 02:16:17.033653 7085 external_mini_cluster.cc:1428] Setting key 97eda8f8592f54fa1b74d5b7d81aed70
I20260928 02:16:17.037488 28337 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:16:17.047521 28340 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:16:17.035438 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.49:56167 (local address 127.0.0.1:46493)
0928 02:16:17.036200 (+ 762us) server_negotiation.cc:206] Beginning negotiation
0928 02:16:17.036214 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0928 02:16:17.036257 (+ 43us) server_negotiation.cc:407] Connection header received
0928 02:16:17.038108 (+ 1851us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:16:17.038135 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:16:17.038571 (+ 436us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:16:17.039000 (+ 429us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:16:17.039571 (+ 571us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:16:17.041021 (+ 1450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:16:17.042474 (+ 1453us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:16:17.043236 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:16:17.043451 (+ 215us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:16:17.044977 (+ 1526us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:16:17.044989 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:16:17.045005 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 02:16:17.045584 (+ 579us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:16:17.045668 (+ 84us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 02:16:17.045697 (+ 29us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:16:17.045927 (+ 230us) server_negotiation.cc:1035] Waiting for connection context
0928 02:16:17.046161 (+ 234us) server_negotiation.cc:299] Negotiation successful
0928 02:16:17.046353 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":549,"thread_start_us":280,"threads_started":1}
I20260928 02:16:17.049475 28337 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f0d89bccc01e45b8b4362012f52a5d7b: Bootstrap starting.
I20260928 02:16:17.051751 7085 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 02:16:17.053149 28337 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f0d89bccc01e45b8b4362012f52a5d7b: Neither blocks nor log segments found. Creating new log.
I20260928 02:16:17.054234 28337 log.cc:824] T 00000000000000000000000000000000 P f0d89bccc01e45b8b4362012f52a5d7b: Log is configured to *not* fsync() on all Append() calls
I20260928 02:16:17.058043 28337 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f0d89bccc01e45b8b4362012f52a5d7b: No bootstrap required, opened a new log
I20260928 02:16:17.062978 28340 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:16:17.056524 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.49:46215 (local address 127.0.0.1:46493)
0928 02:16:17.056732 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0928 02:16:17.056741 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 02:16:17.057198 (+ 457us) server_negotiation.cc:407] Connection header received
0928 02:16:17.057434 (+ 236us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:16:17.057438 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:16:17.057502 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:16:17.057607 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:16:17.058132 (+ 525us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:16:17.058962 (+ 830us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:16:17.060293 (+ 1331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:16:17.060781 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:16:17.060975 (+ 194us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:16:17.062174 (+ 1199us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:16:17.062179 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:16:17.062182 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 02:16:17.062201 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:16:17.062243 (+ 42us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 02:16:17.062261 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:16:17.062406 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0928 02:16:17.062633 (+ 227us) server_negotiation.cc:299] Negotiation successful
0928 02:16:17.062694 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260928 02:16:17.066596 28337 raft_consensus.cc:348] T 00000000000000000000000000000000 P f0d89bccc01e45b8b4362012f52a5d7b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f0d89bccc01e45b8b4362012f52a5d7b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46493 } }
I20260928 02:16:17.067051 28337 raft_consensus.cc:374] T 00000000000000000000000000000000 P f0d89bccc01e45b8b4362012f52a5d7b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:16:17.067163 28337 raft_consensus.cc:729] T 00000000000000000000000000000000 P f0d89bccc01e45b8b4362012f52a5d7b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f0d89bccc01e45b8b4362012f52a5d7b, State: Initialized, Role: FOLLOWER
I20260928 02:16:17.067812 28337 consensus_queue.cc:260] T 00000000000000000000000000000000 P f0d89bccc01e45b8b4362012f52a5d7b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f0d89bccc01e45b8b4362012f52a5d7b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46493 } }
I20260928 02:16:17.068068 28337 raft_consensus.cc:388] T 00000000000000000000000000000000 P f0d89bccc01e45b8b4362012f52a5d7b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 02:16:17.068224 28337 raft_consensus.cc:482] T 00000000000000000000000000000000 P f0d89bccc01e45b8b4362012f52a5d7b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 02:16:17.068418 28337 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f0d89bccc01e45b8b4362012f52a5d7b [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:16:17.071169 28337 raft_consensus.cc:504] T 00000000000000000000000000000000 P f0d89bccc01e45b8b4362012f52a5d7b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f0d89bccc01e45b8b4362012f52a5d7b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46493 } }
I20260928 02:16:17.071831 28337 leader_election.cc:302] T 00000000000000000000000000000000 P f0d89bccc01e45b8b4362012f52a5d7b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f0d89bccc01e45b8b4362012f52a5d7b; no voters:
I20260928 02:16:17.072432 28337 leader_election.cc:288] T 00000000000000000000000000000000 P f0d89bccc01e45b8b4362012f52a5d7b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 02:16:17.072572 28347 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f0d89bccc01e45b8b4362012f52a5d7b [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:16:17.073408 28347 raft_consensus.cc:686] T 00000000000000000000000000000000 P f0d89bccc01e45b8b4362012f52a5d7b [term 1 LEADER]: Becoming Leader. State: Replica: f0d89bccc01e45b8b4362012f52a5d7b, State: Running, Role: LEADER
I20260928 02:16:17.074010 28347 consensus_queue.cc:237] T 00000000000000000000000000000000 P f0d89bccc01e45b8b4362012f52a5d7b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f0d89bccc01e45b8b4362012f52a5d7b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46493 } }
I20260928 02:16:17.074724 28337 sys_catalog.cc:564] T 00000000000000000000000000000000 P f0d89bccc01e45b8b4362012f52a5d7b [sys.catalog]: configured and running, proceeding with master startup.
I20260928 02:16:17.076938 28348 sys_catalog.cc:455] T 00000000000000000000000000000000 P f0d89bccc01e45b8b4362012f52a5d7b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f0d89bccc01e45b8b4362012f52a5d7b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f0d89bccc01e45b8b4362012f52a5d7b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46493 } } }
I20260928 02:16:17.077446 28348 sys_catalog.cc:458] T 00000000000000000000000000000000 P f0d89bccc01e45b8b4362012f52a5d7b [sys.catalog]: This master's current role is: LEADER
I20260928 02:16:17.077118 28349 sys_catalog.cc:455] T 00000000000000000000000000000000 P f0d89bccc01e45b8b4362012f52a5d7b [sys.catalog]: SysCatalogTable state changed. Reason: New leader f0d89bccc01e45b8b4362012f52a5d7b. Latest consensus state: current_term: 1 leader_uuid: "f0d89bccc01e45b8b4362012f52a5d7b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f0d89bccc01e45b8b4362012f52a5d7b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46493 } } }
I20260928 02:16:17.078492 28349 sys_catalog.cc:458] T 00000000000000000000000000000000 P f0d89bccc01e45b8b4362012f52a5d7b [sys.catalog]: This master's current role is: LEADER
I20260928 02:16:17.079830 28353 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 02:16:17.085592 28353 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 02:16:17.096588 28353 catalog_manager.cc:1347] Generated new cluster ID: 60984ae77fab4168bf0a73822a6b55f8
I20260928 02:16:17.096750 28353 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 02:16:17.115806 28353 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 02:16:17.117375 28353 catalog_manager.cc:1504] Loading token signing keys...
I20260928 02:16:17.125496 28353 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f0d89bccc01e45b8b4362012f52a5d7b: Generated new TSK 0
I20260928 02:16:17.126709 28353 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 02:16:17.143540 7085 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskTlaUuc/build/asan/bin/kudu pid 28270
W20260928 02:16:17.144742 28332 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.49:56167
W20260928 02:16:17.145797 7085 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskTlaUuc/build/asan/bin/kudu pid 28270: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260928 02:16:17.145967 7085 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTlaUuc/build/asan/bin/kudu with pid 28270
W20260928 02:16:17.164279 28255 connection.cc:535] client connection to 127.0.0.1:46493 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:46493): Connection reset by peer (error 104)
2026-09-28T02:16:17Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/7 (1993 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/8
Loading random data
Initializing database '/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790561432251112-7085-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 02:16:17 dist-test-slave-86gv krb5kdc[28368](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 02:16:17 dist-test-slave-86gv krb5kdc[28368](info): set up 2 sockets
Sep 28 02:16:17 dist-test-slave-86gv krb5kdc[28368](info): commencing operation
krb5kdc: starting...
W20260928 02:16:18.270017 7085 mini_kdc.cc:121] Time spent starting KDC: real 1.086s user 0.000s sys 0.030s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790561432251112-7085-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790561432251112-7085-0/krb5kdc/test-user.keytab.
Sep 28 02:16:18 dist-test-slave-86gv krb5kdc[28368](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561778, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T02:16:18Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T02:16:18Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790561432251112-7085-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790561432251112-7085-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 02:16:18.556419 7085 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTlaUuc/build/asan/bin/kudu
/tmp/dist-test-taskTlaUuc/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790561432251112-7085-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790561432251112-7085-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790561432251112-7085-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790561432251112-7085-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38175
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790561432251112-7085-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:35401
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:38175
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790561432251112-7085-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790561432251112-7085-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790561432251112-7085-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:16:18.912364 28381 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:16:18.912676 28381 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:16:18.912858 28381 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:16:18.920555 28381 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 02:16:18.920701 28381 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:16:18.920915 28381 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:16:18.921047 28381 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 02:16:18.921140 28381 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 02:16:18.932341 28381 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:35401
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790561432251112-7085-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790561432251112-7085-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:38175
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790561432251112-7085-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38175
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790561432251112-7085-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790561432251112-7085-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 00:51:59 UTC on ccycloud.quasar-axgkwn.root.comops.site
build id 4676464
ASAN enabled
I20260928 02:16:18.933934 28381 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:16:18.935413 28381 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:16:18.948285 28386 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:16:18.948185 28389 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:16:18.948251 28387 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:16:18.948549 28381 server_base.cc:1034] running on GCE node
I20260928 02:16:18.949798 28381 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:16:18.951282 28381 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:16:18.952499 28381 hybrid_clock.cc:648] HybridClock initialized: now 1790561778952433 us; error 58 us; skew 500 ppm
I20260928 02:16:18.952821 28381 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 02:16:18 dist-test-slave-86gv krb5kdc[28368](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561778, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:16:18.961548 28381 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 02:16:18.963264 28381 webserver.cc:460] Webserver started at http://127.0.0.1:33737/ using document root <none> and password file <none>
I20260928 02:16:18.964048 28381 fs_manager.cc:359] Metadata directory not provided
I20260928 02:16:18.964185 28381 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:16:18.964540 28381 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:16:18.967803 28381 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790561432251112-7085-0/minicluster-data/master-0/data/instance:
uuid: "c80ac8324e324103975c1f158a48c78a"
format_stamp: "Formatted at 2026-09-28 02:16:18 on dist-test-slave-86gv"
server_key: "3d779115e38a2f5a2ccc7607dc3d4d13"
server_key_iv: "96241680244196bc0bd75465077ae847"
server_key_version: "encryptionkey@0"
I20260928 02:16:18.968685 28381 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790561432251112-7085-0/minicluster-data/master-0/wal/instance:
uuid: "c80ac8324e324103975c1f158a48c78a"
format_stamp: "Formatted at 2026-09-28 02:16:18 on dist-test-slave-86gv"
server_key: "3d779115e38a2f5a2ccc7607dc3d4d13"
server_key_iv: "96241680244196bc0bd75465077ae847"
server_key_version: "encryptionkey@0"
I20260928 02:16:18.974531 28381 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260928 02:16:18.978389 28396 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:16:18.980062 28381 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260928 02:16:18.980285 28381 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790561432251112-7085-0/minicluster-data/master-0/data,/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790561432251112-7085-0/minicluster-data/master-0/wal
uuid: "c80ac8324e324103975c1f158a48c78a"
format_stamp: "Formatted at 2026-09-28 02:16:18 on dist-test-slave-86gv"
server_key: "3d779115e38a2f5a2ccc7607dc3d4d13"
server_key_iv: "96241680244196bc0bd75465077ae847"
server_key_version: "encryptionkey@0"
I20260928 02:16:18.980487 28381 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790561432251112-7085-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790561432251112-7085-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790561432251112-7085-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:16:18.987047 28381 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:16:18.992053 28381 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:16:18.992388 28381 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:16:19.012707 28381 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:38175
I20260928 02:16:19.012746 28447 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:38175 every 8 connection(s)
I20260928 02:16:19.014310 28381 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790561432251112-7085-0/minicluster-data/master-0/data/info.pb
I20260928 02:16:19.014928 7085 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTlaUuc/build/asan/bin/kudu as pid 28381
I20260928 02:16:19.015151 7085 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790561432251112-7085-0/minicluster-data/master-0/wal/instance
I20260928 02:16:19.015776 7085 external_mini_cluster.cc:1428] Setting key 175dbb3fc9a0057006e65c2df6176739
I20260928 02:16:19.019354 28448 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:16:19.025051 28451 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:16:19.017871 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.49:39757 (local address 127.0.0.1:38175)
0928 02:16:19.018749 (+ 878us) server_negotiation.cc:206] Beginning negotiation
0928 02:16:19.018768 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0928 02:16:19.018823 (+ 55us) server_negotiation.cc:407] Connection header received
0928 02:16:19.020762 (+ 1939us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:16:19.020790 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:16:19.021296 (+ 506us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:16:19.021750 (+ 454us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:16:19.022981 (+ 1231us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:16:19.022987 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:16:19.023002 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 02:16:19.023031 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:16:19.023110 (+ 79us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 02:16:19.023140 (+ 30us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:16:19.023446 (+ 306us) server_negotiation.cc:1035] Waiting for connection context
0928 02:16:19.023615 (+ 169us) server_negotiation.cc:299] Negotiation successful
0928 02:16:19.023870 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":604,"thread_start_us":363,"threads_started":1}
I20260928 02:16:19.029448 7085 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 02:16:19.032006 28448 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c80ac8324e324103975c1f158a48c78a: Bootstrap starting.
I20260928 02:16:19.036295 28448 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c80ac8324e324103975c1f158a48c78a: Neither blocks nor log segments found. Creating new log.
I20260928 02:16:19.037015 28451 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:16:19.034190 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.49:48905 (local address 127.0.0.1:38175)
0928 02:16:19.034442 (+ 252us) server_negotiation.cc:206] Beginning negotiation
0928 02:16:19.034451 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 02:16:19.034784 (+ 333us) server_negotiation.cc:407] Connection header received
0928 02:16:19.035052 (+ 268us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:16:19.035057 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:16:19.035110 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:16:19.035220 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:16:19.036230 (+ 1010us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:16:19.036234 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:16:19.036238 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 02:16:19.036254 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:16:19.036299 (+ 45us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 02:16:19.036316 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:16:19.036472 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0928 02:16:19.036584 (+ 112us) server_negotiation.cc:299] Negotiation successful
0928 02:16:19.036662 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260928 02:16:19.037744 28448 log.cc:824] T 00000000000000000000000000000000 P c80ac8324e324103975c1f158a48c78a: Log is configured to *not* fsync() on all Append() calls
I20260928 02:16:19.040838 28448 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c80ac8324e324103975c1f158a48c78a: No bootstrap required, opened a new log
I20260928 02:16:19.048365 28448 raft_consensus.cc:348] T 00000000000000000000000000000000 P c80ac8324e324103975c1f158a48c78a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c80ac8324e324103975c1f158a48c78a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38175 } }
I20260928 02:16:19.048835 28448 raft_consensus.cc:374] T 00000000000000000000000000000000 P c80ac8324e324103975c1f158a48c78a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:16:19.048995 28448 raft_consensus.cc:729] T 00000000000000000000000000000000 P c80ac8324e324103975c1f158a48c78a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c80ac8324e324103975c1f158a48c78a, State: Initialized, Role: FOLLOWER
I20260928 02:16:19.049710 28448 consensus_queue.cc:260] T 00000000000000000000000000000000 P c80ac8324e324103975c1f158a48c78a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c80ac8324e324103975c1f158a48c78a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38175 } }
I20260928 02:16:19.049969 28448 raft_consensus.cc:388] T 00000000000000000000000000000000 P c80ac8324e324103975c1f158a48c78a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 02:16:19.050123 28448 raft_consensus.cc:482] T 00000000000000000000000000000000 P c80ac8324e324103975c1f158a48c78a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 02:16:19.050319 28448 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c80ac8324e324103975c1f158a48c78a [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:16:19.052668 28448 raft_consensus.cc:504] T 00000000000000000000000000000000 P c80ac8324e324103975c1f158a48c78a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c80ac8324e324103975c1f158a48c78a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38175 } }
I20260928 02:16:19.053315 28448 leader_election.cc:302] T 00000000000000000000000000000000 P c80ac8324e324103975c1f158a48c78a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c80ac8324e324103975c1f158a48c78a; no voters:
I20260928 02:16:19.053936 28448 leader_election.cc:288] T 00000000000000000000000000000000 P c80ac8324e324103975c1f158a48c78a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 02:16:19.054097 28458 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c80ac8324e324103975c1f158a48c78a [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:16:19.054847 28458 raft_consensus.cc:686] T 00000000000000000000000000000000 P c80ac8324e324103975c1f158a48c78a [term 1 LEADER]: Becoming Leader. State: Replica: c80ac8324e324103975c1f158a48c78a, State: Running, Role: LEADER
I20260928 02:16:19.055371 28458 consensus_queue.cc:237] T 00000000000000000000000000000000 P c80ac8324e324103975c1f158a48c78a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c80ac8324e324103975c1f158a48c78a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38175 } }
I20260928 02:16:19.056190 28448 sys_catalog.cc:564] T 00000000000000000000000000000000 P c80ac8324e324103975c1f158a48c78a [sys.catalog]: configured and running, proceeding with master startup.
I20260928 02:16:19.058456 28460 sys_catalog.cc:455] T 00000000000000000000000000000000 P c80ac8324e324103975c1f158a48c78a [sys.catalog]: SysCatalogTable state changed. Reason: New leader c80ac8324e324103975c1f158a48c78a. Latest consensus state: current_term: 1 leader_uuid: "c80ac8324e324103975c1f158a48c78a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c80ac8324e324103975c1f158a48c78a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38175 } } }
I20260928 02:16:19.058652 28459 sys_catalog.cc:455] T 00000000000000000000000000000000 P c80ac8324e324103975c1f158a48c78a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c80ac8324e324103975c1f158a48c78a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c80ac8324e324103975c1f158a48c78a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38175 } } }
I20260928 02:16:19.058992 28460 sys_catalog.cc:458] T 00000000000000000000000000000000 P c80ac8324e324103975c1f158a48c78a [sys.catalog]: This master's current role is: LEADER
I20260928 02:16:19.059000 28459 sys_catalog.cc:458] T 00000000000000000000000000000000 P c80ac8324e324103975c1f158a48c78a [sys.catalog]: This master's current role is: LEADER
I20260928 02:16:19.061622 28466 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 02:16:19.067261 28466 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 02:16:19.077442 28466 catalog_manager.cc:1347] Generated new cluster ID: 233e3b4736744bce90f16ed9f4f91470
I20260928 02:16:19.077582 28466 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 02:16:19.107749 28466 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 02:16:19.108306 28466 catalog_manager.cc:1504] Loading token signing keys...
I20260928 02:16:19.132411 28466 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c80ac8324e324103975c1f158a48c78a: Generated new TSK 0
I20260928 02:16:19.133639 28466 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 02:16:19.145706 7085 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskTlaUuc/build/asan/bin/kudu pid 28381
W20260928 02:16:19.147100 28443 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.49:39757
W20260928 02:16:19.147902 7085 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskTlaUuc/build/asan/bin/kudu pid 28381: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260928 02:16:19.148082 7085 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTlaUuc/build/asan/bin/kudu with pid 28381
2026-09-28T02:16:19Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/8 (2002 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/9
Loading random data
Initializing database '/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790561432251112-7085-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 02:16:19 dist-test-slave-86gv krb5kdc[28479](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 02:16:19 dist-test-slave-86gv krb5kdc[28479](info): set up 2 sockets
Sep 28 02:16:19 dist-test-slave-86gv krb5kdc[28479](info): commencing operation
krb5kdc: starting...
W20260928 02:16:20.282099 7085 mini_kdc.cc:121] Time spent starting KDC: real 1.096s user 0.000s sys 0.032s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790561432251112-7085-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790561432251112-7085-0/krb5kdc/test-user.keytab.
Sep 28 02:16:20 dist-test-slave-86gv krb5kdc[28479](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561780, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T02:16:20Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T02:16:20Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790561432251112-7085-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790561432251112-7085-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 02:16:20.569205 7085 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTlaUuc/build/asan/bin/kudu
/tmp/dist-test-taskTlaUuc/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790561432251112-7085-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790561432251112-7085-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790561432251112-7085-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790561432251112-7085-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45753
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790561432251112-7085-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:46165
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:45753
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790561432251112-7085-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790561432251112-7085-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790561432251112-7085-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:16:20.934398 28492 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:16:20.934736 28492 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:16:20.934835 28492 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:16:20.942735 28492 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 02:16:20.942869 28492 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:16:20.942916 28492 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:16:20.942970 28492 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 02:16:20.943014 28492 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 02:16:20.953442 28492 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:46165
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790561432251112-7085-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790561432251112-7085-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:45753
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790561432251112-7085-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45753
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790561432251112-7085-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790561432251112-7085-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 00:51:59 UTC on ccycloud.quasar-axgkwn.root.comops.site
build id 4676464
ASAN enabled
I20260928 02:16:20.954862 28492 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:16:20.956208 28492 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:16:20.968811 28498 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:16:20.968775 28500 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:16:20.968852 28497 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:16:20.969552 28492 server_base.cc:1034] running on GCE node
I20260928 02:16:20.970439 28492 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:16:20.972075 28492 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:16:20.973304 28492 hybrid_clock.cc:648] HybridClock initialized: now 1790561780973245 us; error 49 us; skew 500 ppm
I20260928 02:16:20.973704 28492 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 02:16:20 dist-test-slave-86gv krb5kdc[28479](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561780, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:16:20.986085 28492 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 02:16:20.988281 28492 webserver.cc:460] Webserver started at http://127.0.0.1:41375/ using document root <none> and password file <none>
I20260928 02:16:20.989307 28492 fs_manager.cc:359] Metadata directory not provided
I20260928 02:16:20.989513 28492 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:16:20.989923 28492 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:16:20.993908 28492 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790561432251112-7085-0/minicluster-data/master-0/data/instance:
uuid: "1d230dc765214e72b0248e2c8026e369"
format_stamp: "Formatted at 2026-09-28 02:16:20 on dist-test-slave-86gv"
server_key: "24ad6580fc60d10887456524139984bf"
server_key_iv: "8e2d818ee83ba9fc1186994155901d1f"
server_key_version: "encryptionkey@0"
I20260928 02:16:20.994990 28492 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790561432251112-7085-0/minicluster-data/master-0/wal/instance:
uuid: "1d230dc765214e72b0248e2c8026e369"
format_stamp: "Formatted at 2026-09-28 02:16:20 on dist-test-slave-86gv"
server_key: "24ad6580fc60d10887456524139984bf"
server_key_iv: "8e2d818ee83ba9fc1186994155901d1f"
server_key_version: "encryptionkey@0"
I20260928 02:16:21.002045 28492 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.007s sys 0.001s
I20260928 02:16:21.006212 28507 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:16:21.008011 28492 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20260928 02:16:21.008251 28492 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790561432251112-7085-0/minicluster-data/master-0/data,/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790561432251112-7085-0/minicluster-data/master-0/wal
uuid: "1d230dc765214e72b0248e2c8026e369"
format_stamp: "Formatted at 2026-09-28 02:16:20 on dist-test-slave-86gv"
server_key: "24ad6580fc60d10887456524139984bf"
server_key_iv: "8e2d818ee83ba9fc1186994155901d1f"
server_key_version: "encryptionkey@0"
I20260928 02:16:21.008495 28492 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790561432251112-7085-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790561432251112-7085-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790561432251112-7085-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:16:21.030891 28492 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:16:21.036375 28492 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:16:21.036793 28492 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:16:21.060671 28492 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:45753
I20260928 02:16:21.060689 28558 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:45753 every 8 connection(s)
I20260928 02:16:21.062556 28492 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790561432251112-7085-0/minicluster-data/master-0/data/info.pb
I20260928 02:16:21.067050 28559 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:16:21.068957 7085 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTlaUuc/build/asan/bin/kudu as pid 28492
I20260928 02:16:21.069145 7085 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790561432251112-7085-0/minicluster-data/master-0/wal/instance
I20260928 02:16:21.069675 7085 external_mini_cluster.cc:1428] Setting key 0e874faad64afb22ad6f4f0e39b3ae95
I20260928 02:16:21.081794 28559 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1d230dc765214e72b0248e2c8026e369: Bootstrap starting.
I20260928 02:16:21.085352 28559 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1d230dc765214e72b0248e2c8026e369: Neither blocks nor log segments found. Creating new log.
I20260928 02:16:21.086612 28559 log.cc:824] T 00000000000000000000000000000000 P 1d230dc765214e72b0248e2c8026e369: Log is configured to *not* fsync() on all Append() calls
I20260928 02:16:21.090413 28559 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1d230dc765214e72b0248e2c8026e369: No bootstrap required, opened a new log
Sep 28 02:16:21 dist-test-slave-86gv krb5kdc[28479](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561780, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260928 02:16:21.101960 28562 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:16:21.074649 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.49:53259 (local address 127.0.0.1:45753)
0928 02:16:21.075590 (+ 941us) server_negotiation.cc:206] Beginning negotiation
0928 02:16:21.075605 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0928 02:16:21.075659 (+ 54us) server_negotiation.cc:407] Connection header received
0928 02:16:21.077993 (+ 2334us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:16:21.078025 (+ 32us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:16:21.078632 (+ 607us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:16:21.079226 (+ 594us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:16:21.081291 (+ 2065us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:16:21.083150 (+ 1859us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:16:21.084931 (+ 1781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:16:21.086002 (+ 1071us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:16:21.086267 (+ 265us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:16:21.092131 (+ 5864us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:16:21.092139 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:16:21.092156 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:16:21.092197 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:16:21.095648 (+ 3451us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:16:21.096711 (+ 1063us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:16:21.096723 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:16:21.096736 (+ 13us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:16:21.097036 (+ 300us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:16:21.097682 (+ 646us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:16:21.097689 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:16:21.097694 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:16:21.098629 (+ 935us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:16:21.099008 (+ 379us) server_negotiation.cc:1035] Waiting for connection context
0928 02:16:21.099670 (+ 662us) server_negotiation.cc:299] Negotiation successful
0928 02:16:21.100123 (+ 453us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":678,"thread_start_us":378,"threads_started":1}
I20260928 02:16:21.102473 28559 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1d230dc765214e72b0248e2c8026e369 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1d230dc765214e72b0248e2c8026e369" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45753 } }
I20260928 02:16:21.103082 28559 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1d230dc765214e72b0248e2c8026e369 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:16:21.103237 28559 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1d230dc765214e72b0248e2c8026e369 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1d230dc765214e72b0248e2c8026e369, State: Initialized, Role: FOLLOWER
I20260928 02:16:21.103999 28559 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1d230dc765214e72b0248e2c8026e369 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1d230dc765214e72b0248e2c8026e369" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45753 } }
I20260928 02:16:21.104321 28559 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1d230dc765214e72b0248e2c8026e369 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 02:16:21.104526 28559 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1d230dc765214e72b0248e2c8026e369 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 02:16:21.104736 28559 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1d230dc765214e72b0248e2c8026e369 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:16:21.106894 7085 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 02:16:21.107986 28559 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1d230dc765214e72b0248e2c8026e369 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1d230dc765214e72b0248e2c8026e369" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45753 } }
I20260928 02:16:21.108600 28559 leader_election.cc:302] T 00000000000000000000000000000000 P 1d230dc765214e72b0248e2c8026e369 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1d230dc765214e72b0248e2c8026e369; no voters:
I20260928 02:16:21.109491 28559 leader_election.cc:288] T 00000000000000000000000000000000 P 1d230dc765214e72b0248e2c8026e369 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 02:16:21.109613 28564 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1d230dc765214e72b0248e2c8026e369 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:16:21.112617 28559 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1d230dc765214e72b0248e2c8026e369 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 02:16:21.128513 28564 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1d230dc765214e72b0248e2c8026e369 [term 1 LEADER]: Becoming Leader. State: Replica: 1d230dc765214e72b0248e2c8026e369, State: Running, Role: LEADER
I20260928 02:16:21.129760 28564 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1d230dc765214e72b0248e2c8026e369 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1d230dc765214e72b0248e2c8026e369" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45753 } }
I20260928 02:16:21.134629 28567 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1d230dc765214e72b0248e2c8026e369 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1d230dc765214e72b0248e2c8026e369" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1d230dc765214e72b0248e2c8026e369" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45753 } } }
I20260928 02:16:21.135273 28567 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1d230dc765214e72b0248e2c8026e369 [sys.catalog]: This master's current role is: LEADER
I20260928 02:16:21.140300 28580 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 02:16:21.140319 28568 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1d230dc765214e72b0248e2c8026e369 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1d230dc765214e72b0248e2c8026e369. Latest consensus state: current_term: 1 leader_uuid: "1d230dc765214e72b0248e2c8026e369" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1d230dc765214e72b0248e2c8026e369" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45753 } } }
I20260928 02:16:21.140790 28568 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1d230dc765214e72b0248e2c8026e369 [sys.catalog]: This master's current role is: LEADER
I20260928 02:16:21.149294 28580 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 02:16:21.150208 28562 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:16:21.127756 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.49:49875 (local address 127.0.0.1:45753)
0928 02:16:21.128011 (+ 255us) server_negotiation.cc:206] Beginning negotiation
0928 02:16:21.128020 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 02:16:21.129048 (+ 1028us) server_negotiation.cc:407] Connection header received
0928 02:16:21.129420 (+ 372us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:16:21.129428 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:16:21.129527 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:16:21.129604 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:16:21.132067 (+ 2463us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:16:21.133380 (+ 1313us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:16:21.135395 (+ 2015us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:16:21.136161 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:16:21.136294 (+ 133us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:16:21.140626 (+ 4332us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:16:21.140640 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:16:21.140647 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:16:21.140696 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:16:21.145621 (+ 4925us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:16:21.146934 (+ 1313us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:16:21.146943 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:16:21.146948 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:16:21.147126 (+ 178us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:16:21.147886 (+ 760us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:16:21.147892 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:16:21.147896 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:16:21.148352 (+ 456us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:16:21.148612 (+ 260us) server_negotiation.cc:1035] Waiting for connection context
0928 02:16:21.149549 (+ 937us) server_negotiation.cc:299] Negotiation successful
0928 02:16:21.149784 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
I20260928 02:16:21.164458 28580 catalog_manager.cc:1347] Generated new cluster ID: a2da983a39104c15ad0dd4dd6c5df0d0
I20260928 02:16:21.164618 28580 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 02:16:21.185786 28580 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 02:16:21.188505 28580 catalog_manager.cc:1504] Loading token signing keys...
I20260928 02:16:21.198299 28580 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1d230dc765214e72b0248e2c8026e369: Generated new TSK 0
I20260928 02:16:21.199405 28580 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 02:16:21.209649 7085 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskTlaUuc/build/asan/bin/kudu pid 28492
I20260928 02:16:21.210990 28554 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 02:16:21.304445 7085 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTlaUuc/build/asan/bin/kudu with pid 28492
W20260928 02:16:21.322762 28477 connection.cc:535] client connection to 127.0.0.1:45753 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:45753): Connection reset by peer (error 104)
2026-09-28T02:16:21Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/9 (2156 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/10
Loading random data
Initializing database '/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790561432251112-7085-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 02:16:21 dist-test-slave-86gv krb5kdc[28592](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 02:16:21 dist-test-slave-86gv krb5kdc[28592](info): set up 2 sockets
Sep 28 02:16:21 dist-test-slave-86gv krb5kdc[28592](info): commencing operation
krb5kdc: starting...
W20260928 02:16:22.425868 7085 mini_kdc.cc:121] Time spent starting KDC: real 1.083s user 0.000s sys 0.029s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790561432251112-7085-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790561432251112-7085-0/krb5kdc/test-user.keytab.
Sep 28 02:16:22 dist-test-slave-86gv krb5kdc[28592](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561782, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T02:16:22Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T02:16:22Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790561432251112-7085-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790561432251112-7085-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 02:16:22.711783 7085 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTlaUuc/build/asan/bin/kudu
/tmp/dist-test-taskTlaUuc/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790561432251112-7085-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790561432251112-7085-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790561432251112-7085-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790561432251112-7085-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36035
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790561432251112-7085-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:35451
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:36035
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790561432251112-7085-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790561432251112-7085-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790561432251112-7085-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:16:23.059582 28605 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:16:23.059902 28605 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:16:23.059996 28605 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:16:23.067461 28605 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 02:16:23.067587 28605 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:16:23.067638 28605 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:16:23.067700 28605 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 02:16:23.067747 28605 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 02:16:23.078318 28605 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:35451
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790561432251112-7085-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790561432251112-7085-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:36035
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790561432251112-7085-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36035
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790561432251112-7085-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790561432251112-7085-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 00:51:59 UTC on ccycloud.quasar-axgkwn.root.comops.site
build id 4676464
ASAN enabled
I20260928 02:16:23.079715 28605 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:16:23.081048 28605 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:16:23.093917 28610 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:16:23.093969 28613 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:16:23.094313 28611 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:16:23.094887 28605 server_base.cc:1034] running on GCE node
I20260928 02:16:23.095695 28605 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:16:23.097301 28605 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:16:23.098523 28605 hybrid_clock.cc:648] HybridClock initialized: now 1790561783098484 us; error 28 us; skew 500 ppm
I20260928 02:16:23.098909 28605 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 02:16:23 dist-test-slave-86gv krb5kdc[28592](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561783, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:16:23.102434 28605 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 02:16:23.103972 28605 webserver.cc:460] Webserver started at http://127.0.0.1:40567/ using document root <none> and password file <none>
I20260928 02:16:23.104683 28605 fs_manager.cc:359] Metadata directory not provided
I20260928 02:16:23.104790 28605 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:16:23.105134 28605 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:16:23.108372 28605 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790561432251112-7085-0/minicluster-data/master-0/data/instance:
uuid: "08ed83ef5ce34b56b38aa517edd70dc4"
format_stamp: "Formatted at 2026-09-28 02:16:23 on dist-test-slave-86gv"
server_key: "fe2c0ee3e0d3a79eaf049762292e27c0"
server_key_iv: "5cb9de9d57c07ef8a75ec6c275ef0db7"
server_key_version: "encryptionkey@0"
I20260928 02:16:23.109210 28605 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790561432251112-7085-0/minicluster-data/master-0/wal/instance:
uuid: "08ed83ef5ce34b56b38aa517edd70dc4"
format_stamp: "Formatted at 2026-09-28 02:16:23 on dist-test-slave-86gv"
server_key: "fe2c0ee3e0d3a79eaf049762292e27c0"
server_key_iv: "5cb9de9d57c07ef8a75ec6c275ef0db7"
server_key_version: "encryptionkey@0"
I20260928 02:16:23.114878 28605 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260928 02:16:23.118887 28620 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:16:23.120524 28605 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260928 02:16:23.120682 28605 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790561432251112-7085-0/minicluster-data/master-0/data,/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790561432251112-7085-0/minicluster-data/master-0/wal
uuid: "08ed83ef5ce34b56b38aa517edd70dc4"
format_stamp: "Formatted at 2026-09-28 02:16:23 on dist-test-slave-86gv"
server_key: "fe2c0ee3e0d3a79eaf049762292e27c0"
server_key_iv: "5cb9de9d57c07ef8a75ec6c275ef0db7"
server_key_version: "encryptionkey@0"
I20260928 02:16:23.120872 28605 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790561432251112-7085-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790561432251112-7085-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790561432251112-7085-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:16:23.140306 28605 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:16:23.145115 28605 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:16:23.145367 28605 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:16:23.164582 28605 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:36035
I20260928 02:16:23.164609 28671 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:36035 every 8 connection(s)
I20260928 02:16:23.166149 28605 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790561432251112-7085-0/minicluster-data/master-0/data/info.pb
I20260928 02:16:23.168833 7085 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTlaUuc/build/asan/bin/kudu as pid 28605
I20260928 02:16:23.169050 7085 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790561432251112-7085-0/minicluster-data/master-0/wal/instance
I20260928 02:16:23.169520 7085 external_mini_cluster.cc:1428] Setting key d40624c9caf98db4852ebd4803040dea
I20260928 02:16:23.170490 28672 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:16:23.185505 28675 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:16:23.171572 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.49:45155 (local address 127.0.0.1:36035)
0928 02:16:23.172368 (+ 796us) server_negotiation.cc:206] Beginning negotiation
0928 02:16:23.172384 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0928 02:16:23.172437 (+ 53us) server_negotiation.cc:407] Connection header received
0928 02:16:23.174319 (+ 1882us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:16:23.174349 (+ 30us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:16:23.174952 (+ 603us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:16:23.175430 (+ 478us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:16:23.175981 (+ 551us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:16:23.178080 (+ 2099us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:16:23.179484 (+ 1404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:16:23.180593 (+ 1109us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:16:23.180795 (+ 202us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:16:23.182193 (+ 1398us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:16:23.182205 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:16:23.182227 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 02:16:23.183063 (+ 836us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:16:23.183172 (+ 109us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 02:16:23.183216 (+ 44us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:16:23.183391 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0928 02:16:23.183765 (+ 374us) server_negotiation.cc:299] Negotiation successful
0928 02:16:23.183998 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":559,"thread_start_us":326,"threads_started":1}
I20260928 02:16:23.186695 28672 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 08ed83ef5ce34b56b38aa517edd70dc4: Bootstrap starting.
I20260928 02:16:23.190266 7085 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 02:16:23.190848 28672 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 08ed83ef5ce34b56b38aa517edd70dc4: Neither blocks nor log segments found. Creating new log.
I20260928 02:16:23.192318 28672 log.cc:824] T 00000000000000000000000000000000 P 08ed83ef5ce34b56b38aa517edd70dc4: Log is configured to *not* fsync() on all Append() calls
I20260928 02:16:23.195364 28672 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 08ed83ef5ce34b56b38aa517edd70dc4: No bootstrap required, opened a new log
I20260928 02:16:23.203025 28675 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:16:23.195609 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.49:54547 (local address 127.0.0.1:36035)
0928 02:16:23.195831 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0928 02:16:23.195841 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 02:16:23.196411 (+ 570us) server_negotiation.cc:407] Connection header received
0928 02:16:23.196620 (+ 209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:16:23.196624 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:16:23.196687 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:16:23.196765 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:16:23.197452 (+ 687us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:16:23.198274 (+ 822us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:16:23.200180 (+ 1906us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:16:23.200635 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:16:23.200740 (+ 105us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:16:23.202176 (+ 1436us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:16:23.202181 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:16:23.202184 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 02:16:23.202207 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:16:23.202244 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 02:16:23.202264 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:16:23.202356 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0928 02:16:23.202582 (+ 226us) server_negotiation.cc:299] Negotiation successful
0928 02:16:23.202628 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260928 02:16:23.205636 28672 raft_consensus.cc:348] T 00000000000000000000000000000000 P 08ed83ef5ce34b56b38aa517edd70dc4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "08ed83ef5ce34b56b38aa517edd70dc4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36035 } }
I20260928 02:16:23.206066 28672 raft_consensus.cc:374] T 00000000000000000000000000000000 P 08ed83ef5ce34b56b38aa517edd70dc4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:16:23.206194 28672 raft_consensus.cc:729] T 00000000000000000000000000000000 P 08ed83ef5ce34b56b38aa517edd70dc4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 08ed83ef5ce34b56b38aa517edd70dc4, State: Initialized, Role: FOLLOWER
I20260928 02:16:23.206804 28672 consensus_queue.cc:260] T 00000000000000000000000000000000 P 08ed83ef5ce34b56b38aa517edd70dc4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "08ed83ef5ce34b56b38aa517edd70dc4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36035 } }
I20260928 02:16:23.207048 28672 raft_consensus.cc:388] T 00000000000000000000000000000000 P 08ed83ef5ce34b56b38aa517edd70dc4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 02:16:23.207211 28672 raft_consensus.cc:482] T 00000000000000000000000000000000 P 08ed83ef5ce34b56b38aa517edd70dc4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 02:16:23.207397 28672 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 08ed83ef5ce34b56b38aa517edd70dc4 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:16:23.209774 28672 raft_consensus.cc:504] T 00000000000000000000000000000000 P 08ed83ef5ce34b56b38aa517edd70dc4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "08ed83ef5ce34b56b38aa517edd70dc4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36035 } }
I20260928 02:16:23.210420 28672 leader_election.cc:302] T 00000000000000000000000000000000 P 08ed83ef5ce34b56b38aa517edd70dc4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 08ed83ef5ce34b56b38aa517edd70dc4; no voters:
I20260928 02:16:23.211154 28672 leader_election.cc:288] T 00000000000000000000000000000000 P 08ed83ef5ce34b56b38aa517edd70dc4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 02:16:23.211288 28682 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 08ed83ef5ce34b56b38aa517edd70dc4 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:16:23.212172 28682 raft_consensus.cc:686] T 00000000000000000000000000000000 P 08ed83ef5ce34b56b38aa517edd70dc4 [term 1 LEADER]: Becoming Leader. State: Replica: 08ed83ef5ce34b56b38aa517edd70dc4, State: Running, Role: LEADER
I20260928 02:16:23.212736 28682 consensus_queue.cc:237] T 00000000000000000000000000000000 P 08ed83ef5ce34b56b38aa517edd70dc4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "08ed83ef5ce34b56b38aa517edd70dc4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36035 } }
I20260928 02:16:23.213593 28672 sys_catalog.cc:564] T 00000000000000000000000000000000 P 08ed83ef5ce34b56b38aa517edd70dc4 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 02:16:23.215718 28683 sys_catalog.cc:455] T 00000000000000000000000000000000 P 08ed83ef5ce34b56b38aa517edd70dc4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "08ed83ef5ce34b56b38aa517edd70dc4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "08ed83ef5ce34b56b38aa517edd70dc4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36035 } } }
I20260928 02:16:23.216104 28683 sys_catalog.cc:458] T 00000000000000000000000000000000 P 08ed83ef5ce34b56b38aa517edd70dc4 [sys.catalog]: This master's current role is: LEADER
I20260928 02:16:23.215829 28684 sys_catalog.cc:455] T 00000000000000000000000000000000 P 08ed83ef5ce34b56b38aa517edd70dc4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 08ed83ef5ce34b56b38aa517edd70dc4. Latest consensus state: current_term: 1 leader_uuid: "08ed83ef5ce34b56b38aa517edd70dc4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "08ed83ef5ce34b56b38aa517edd70dc4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36035 } } }
I20260928 02:16:23.216290 28684 sys_catalog.cc:458] T 00000000000000000000000000000000 P 08ed83ef5ce34b56b38aa517edd70dc4 [sys.catalog]: This master's current role is: LEADER
I20260928 02:16:23.218636 28687 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 02:16:23.224346 28687 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 02:16:23.234835 28687 catalog_manager.cc:1347] Generated new cluster ID: e8956448a08f48b5a9afabb5352eabf3
I20260928 02:16:23.234989 28687 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 02:16:23.253348 28687 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 02:16:23.254806 28687 catalog_manager.cc:1504] Loading token signing keys...
I20260928 02:16:23.265761 28687 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 08ed83ef5ce34b56b38aa517edd70dc4: Generated new TSK 0
I20260928 02:16:23.266772 28687 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 02:16:23.276225 7085 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskTlaUuc/build/asan/bin/kudu pid 28605
W20260928 02:16:23.277882 28667 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.49:45155
W20260928 02:16:23.278659 7085 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskTlaUuc/build/asan/bin/kudu pid 28605: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260928 02:16:23.278827 7085 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTlaUuc/build/asan/bin/kudu with pid 28605
W20260928 02:16:23.299039 28590 connection.cc:535] client connection to 127.0.0.1:36035 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:36035): Connection reset by peer (error 104)
2026-09-28T02:16:23Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/10 (1976 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/11
Loading random data
Initializing database '/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790561432251112-7085-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 02:16:23 dist-test-slave-86gv krb5kdc[28703](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 02:16:23 dist-test-slave-86gv krb5kdc[28703](info): set up 2 sockets
Sep 28 02:16:23 dist-test-slave-86gv krb5kdc[28703](info): commencing operation
krb5kdc: starting...
W20260928 02:16:24.425817 7085 mini_kdc.cc:121] Time spent starting KDC: real 1.107s user 0.005s sys 0.031s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790561432251112-7085-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790561432251112-7085-0/krb5kdc/test-user.keytab.
Sep 28 02:16:24 dist-test-slave-86gv krb5kdc[28703](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561784, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T02:16:24Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T02:16:24Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790561432251112-7085-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790561432251112-7085-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 02:16:24.708590 7085 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTlaUuc/build/asan/bin/kudu
/tmp/dist-test-taskTlaUuc/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790561432251112-7085-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790561432251112-7085-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790561432251112-7085-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790561432251112-7085-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44853
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790561432251112-7085-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:46259
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:44853
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790561432251112-7085-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790561432251112-7085-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790561432251112-7085-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:16:25.049913 28716 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:16:25.050225 28716 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:16:25.050428 28716 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:16:25.057816 28716 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 02:16:25.057933 28716 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:16:25.058053 28716 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:16:25.058166 28716 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 02:16:25.058254 28716 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 02:16:25.068324 28716 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:46259
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790561432251112-7085-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790561432251112-7085-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:44853
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790561432251112-7085-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44853
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790561432251112-7085-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790561432251112-7085-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 00:51:59 UTC on ccycloud.quasar-axgkwn.root.comops.site
build id 4676464
ASAN enabled
I20260928 02:16:25.069862 28716 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:16:25.071300 28716 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:16:25.083802 28722 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:16:25.083882 28724 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:16:25.083854 28721 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:16:25.084700 28716 server_base.cc:1034] running on GCE node
I20260928 02:16:25.085573 28716 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:16:25.087127 28716 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:16:25.088344 28716 hybrid_clock.cc:648] HybridClock initialized: now 1790561785088302 us; error 33 us; skew 500 ppm
I20260928 02:16:25.088708 28716 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 02:16:25 dist-test-slave-86gv krb5kdc[28703](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561785, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:16:25.101183 28716 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 02:16:25.102891 28716 webserver.cc:460] Webserver started at http://127.0.0.1:38713/ using document root <none> and password file <none>
I20260928 02:16:25.103556 28716 fs_manager.cc:359] Metadata directory not provided
I20260928 02:16:25.103657 28716 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:16:25.103948 28716 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:16:25.107218 28716 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790561432251112-7085-0/minicluster-data/master-0/data/instance:
uuid: "2fe64e084ecb4e55987fb1d833a17635"
format_stamp: "Formatted at 2026-09-28 02:16:25 on dist-test-slave-86gv"
server_key: "b3d44b54701a9c592c746bc6c7404a7f"
server_key_iv: "cfa3ace5fe705958aa4266c976f37343"
server_key_version: "encryptionkey@0"
I20260928 02:16:25.108029 28716 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790561432251112-7085-0/minicluster-data/master-0/wal/instance:
uuid: "2fe64e084ecb4e55987fb1d833a17635"
format_stamp: "Formatted at 2026-09-28 02:16:25 on dist-test-slave-86gv"
server_key: "b3d44b54701a9c592c746bc6c7404a7f"
server_key_iv: "cfa3ace5fe705958aa4266c976f37343"
server_key_version: "encryptionkey@0"
I20260928 02:16:25.113531 28716 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.005s
I20260928 02:16:25.117100 28731 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:16:25.118491 28716 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260928 02:16:25.118685 28716 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790561432251112-7085-0/minicluster-data/master-0/data,/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790561432251112-7085-0/minicluster-data/master-0/wal
uuid: "2fe64e084ecb4e55987fb1d833a17635"
format_stamp: "Formatted at 2026-09-28 02:16:25 on dist-test-slave-86gv"
server_key: "b3d44b54701a9c592c746bc6c7404a7f"
server_key_iv: "cfa3ace5fe705958aa4266c976f37343"
server_key_version: "encryptionkey@0"
I20260928 02:16:25.118930 28716 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790561432251112-7085-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790561432251112-7085-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790561432251112-7085-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:16:25.125144 28716 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:16:25.129462 28716 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:16:25.129771 28716 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:16:25.149408 28716 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:44853
I20260928 02:16:25.149456 28782 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:44853 every 8 connection(s)
I20260928 02:16:25.150898 28716 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790561432251112-7085-0/minicluster-data/master-0/data/info.pb
I20260928 02:16:25.154830 28783 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:16:25.155298 7085 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTlaUuc/build/asan/bin/kudu as pid 28716
I20260928 02:16:25.155510 7085 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790561432251112-7085-0/minicluster-data/master-0/wal/instance
I20260928 02:16:25.156006 7085 external_mini_cluster.cc:1428] Setting key 99fe617e5a30b673065e41eced6a6055
I20260928 02:16:25.165568 28786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:16:25.157939 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.49:48293 (local address 127.0.0.1:44853)
0928 02:16:25.158731 (+ 792us) server_negotiation.cc:206] Beginning negotiation
0928 02:16:25.158747 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0928 02:16:25.158803 (+ 56us) server_negotiation.cc:407] Connection header received
0928 02:16:25.161674 (+ 2871us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:16:25.161705 (+ 31us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:16:25.162201 (+ 496us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:16:25.162675 (+ 474us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:16:25.163625 (+ 950us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:16:25.163633 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:16:25.163650 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 02:16:25.163667 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:16:25.163733 (+ 66us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 02:16:25.163763 (+ 30us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:16:25.163971 (+ 208us) server_negotiation.cc:1035] Waiting for connection context
0928 02:16:25.164100 (+ 129us) server_negotiation.cc:299] Negotiation successful
0928 02:16:25.164304 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":534,"thread_start_us":322,"threads_started":1}
I20260928 02:16:25.169409 7085 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 02:16:25.169435 28783 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2fe64e084ecb4e55987fb1d833a17635: Bootstrap starting.
I20260928 02:16:25.173408 28783 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2fe64e084ecb4e55987fb1d833a17635: Neither blocks nor log segments found. Creating new log.
I20260928 02:16:25.174768 28783 log.cc:824] T 00000000000000000000000000000000 P 2fe64e084ecb4e55987fb1d833a17635: Log is configured to *not* fsync() on all Append() calls
I20260928 02:16:25.177486 28786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:16:25.174844 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.49:36369 (local address 127.0.0.1:44853)
0928 02:16:25.175021 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0928 02:16:25.175027 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 02:16:25.175392 (+ 365us) server_negotiation.cc:407] Connection header received
0928 02:16:25.175578 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:16:25.175585 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:16:25.175636 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:16:25.175723 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:16:25.176787 (+ 1064us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:16:25.176792 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:16:25.176797 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 02:16:25.176814 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:16:25.176856 (+ 42us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 02:16:25.176875 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:16:25.177013 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0928 02:16:25.177124 (+ 111us) server_negotiation.cc:299] Negotiation successful
0928 02:16:25.177197 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260928 02:16:25.178349 28783 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2fe64e084ecb4e55987fb1d833a17635: No bootstrap required, opened a new log
I20260928 02:16:25.186138 28783 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2fe64e084ecb4e55987fb1d833a17635 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2fe64e084ecb4e55987fb1d833a17635" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44853 } }
I20260928 02:16:25.186554 28783 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2fe64e084ecb4e55987fb1d833a17635 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:16:25.186621 28783 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2fe64e084ecb4e55987fb1d833a17635 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2fe64e084ecb4e55987fb1d833a17635, State: Initialized, Role: FOLLOWER
I20260928 02:16:25.187232 28783 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2fe64e084ecb4e55987fb1d833a17635 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2fe64e084ecb4e55987fb1d833a17635" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44853 } }
I20260928 02:16:25.187443 28783 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2fe64e084ecb4e55987fb1d833a17635 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 02:16:25.187544 28783 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2fe64e084ecb4e55987fb1d833a17635 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 02:16:25.187682 28783 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2fe64e084ecb4e55987fb1d833a17635 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:16:25.189945 28783 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2fe64e084ecb4e55987fb1d833a17635 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2fe64e084ecb4e55987fb1d833a17635" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44853 } }
I20260928 02:16:25.190604 28783 leader_election.cc:302] T 00000000000000000000000000000000 P 2fe64e084ecb4e55987fb1d833a17635 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2fe64e084ecb4e55987fb1d833a17635; no voters:
I20260928 02:16:25.191365 28783 leader_election.cc:288] T 00000000000000000000000000000000 P 2fe64e084ecb4e55987fb1d833a17635 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 02:16:25.191565 28793 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2fe64e084ecb4e55987fb1d833a17635 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:16:25.192749 28793 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2fe64e084ecb4e55987fb1d833a17635 [term 1 LEADER]: Becoming Leader. State: Replica: 2fe64e084ecb4e55987fb1d833a17635, State: Running, Role: LEADER
I20260928 02:16:25.193356 28793 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2fe64e084ecb4e55987fb1d833a17635 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2fe64e084ecb4e55987fb1d833a17635" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44853 } }
I20260928 02:16:25.193732 28783 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2fe64e084ecb4e55987fb1d833a17635 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 02:16:25.197007 28794 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2fe64e084ecb4e55987fb1d833a17635 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2fe64e084ecb4e55987fb1d833a17635" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2fe64e084ecb4e55987fb1d833a17635" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44853 } } }
I20260928 02:16:25.197510 28794 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2fe64e084ecb4e55987fb1d833a17635 [sys.catalog]: This master's current role is: LEADER
I20260928 02:16:25.196998 28795 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2fe64e084ecb4e55987fb1d833a17635 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2fe64e084ecb4e55987fb1d833a17635. Latest consensus state: current_term: 1 leader_uuid: "2fe64e084ecb4e55987fb1d833a17635" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2fe64e084ecb4e55987fb1d833a17635" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44853 } } }
I20260928 02:16:25.199146 28795 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2fe64e084ecb4e55987fb1d833a17635 [sys.catalog]: This master's current role is: LEADER
I20260928 02:16:25.202267 28801 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 02:16:25.209360 28801 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 02:16:25.219523 28801 catalog_manager.cc:1347] Generated new cluster ID: adadc540755c4ebb9fe7ccc2aa21f5c2
I20260928 02:16:25.219655 28801 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 02:16:25.242542 28801 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 02:16:25.243022 28801 catalog_manager.cc:1504] Loading token signing keys...
I20260928 02:16:25.262393 28801 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2fe64e084ecb4e55987fb1d833a17635: Generated new TSK 0
I20260928 02:16:25.263391 28801 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 02:16:25.276504 7085 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskTlaUuc/build/asan/bin/kudu pid 28716
W20260928 02:16:25.277793 28778 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.49:48293
W20260928 02:16:25.278594 7085 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskTlaUuc/build/asan/bin/kudu pid 28716: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260928 02:16:25.278746 7085 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTlaUuc/build/asan/bin/kudu with pid 28716
2026-09-28T02:16:25Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/11 (1997 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/12
Loading random data
Initializing database '/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790561432251112-7085-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 02:16:25 dist-test-slave-86gv krb5kdc[28814](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 02:16:25 dist-test-slave-86gv krb5kdc[28814](info): set up 2 sockets
Sep 28 02:16:25 dist-test-slave-86gv krb5kdc[28814](info): commencing operation
krb5kdc: starting...
W20260928 02:16:26.465863 7085 mini_kdc.cc:121] Time spent starting KDC: real 1.150s user 0.003s sys 0.026s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790561432251112-7085-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790561432251112-7085-0/krb5kdc/test-user.keytab.
Sep 28 02:16:26 dist-test-slave-86gv krb5kdc[28814](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561786, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T02:16:26Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T02:16:26Z Disabled control of system clock
WARNING: no policy specified for kudu/127.6.235.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.235.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.235.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790561432251112-7085-0/krb5kdc/kudu_127.6.235.126.keytab.
Entry for principal kudu/127.6.235.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790561432251112-7085-0/krb5kdc/kudu_127.6.235.126.keytab.
I20260928 02:16:26.748538 7085 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTlaUuc/build/asan/bin/kudu
/tmp/dist-test-taskTlaUuc/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790561432251112-7085-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790561432251112-7085-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790561432251112-7085-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790561432251112-7085-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.235.126:39183
--webserver_interface=127.6.235.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790561432251112-7085-0/krb5kdc/kudu_127.6.235.126.keytab
--principal=kudu/127.6.235.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.235.84:41965
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.235.126:39183
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790561432251112-7085-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790561432251112-7085-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790561432251112-7085-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:16:27.104712 28827 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:16:27.105139 28827 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:16:27.105310 28827 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:16:27.113039 28827 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 02:16:27.113185 28827 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:16:27.113274 28827 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:16:27.113358 28827 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 02:16:27.113427 28827 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 02:16:27.123836 28827 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.235.84:41965
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790561432251112-7085-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790561432251112-7085-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.235.126:39183
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790561432251112-7085-0/krb5kdc/kudu_127.6.235.126.keytab
--principal=kudu/127.6.235.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.235.126:39183
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790561432251112-7085-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.235.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790561432251112-7085-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 00:51:59 UTC on ccycloud.quasar-axgkwn.root.comops.site
build id 4676464
ASAN enabled
I20260928 02:16:27.125342 28827 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:16:27.126704 28827 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:16:27.139459 28832 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:16:27.139477 28833 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:16:27.139430 28835 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:16:27.140555 28827 server_base.cc:1034] running on GCE node
I20260928 02:16:27.141543 28827 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:16:27.143100 28827 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:16:27.144333 28827 hybrid_clock.cc:648] HybridClock initialized: now 1790561787144287 us; error 38 us; skew 500 ppm
I20260928 02:16:27.144690 28827 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 02:16:27 dist-test-slave-86gv krb5kdc[28814](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561787, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.235.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:16:27.148228 28827 init.cc:377] Logged in from keytab as kudu/127.6.235.126@KRBTEST.COM (short username kudu)
I20260928 02:16:27.149945 28827 webserver.cc:460] Webserver started at http://127.6.235.126:35665/ using document root <none> and password file <none>
I20260928 02:16:27.150717 28827 fs_manager.cc:359] Metadata directory not provided
I20260928 02:16:27.150883 28827 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:16:27.151226 28827 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:16:27.154454 28827 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790561432251112-7085-0/minicluster-data/master-0/data/instance:
uuid: "309896d6be1144a184529fb8a1bc751b"
format_stamp: "Formatted at 2026-09-28 02:16:27 on dist-test-slave-86gv"
server_key: "b364768fa2220b872663a0a99924b147"
server_key_iv: "c32470bc1a09c13690c84687fc9e99f8"
server_key_version: "encryptionkey@0"
I20260928 02:16:27.155270 28827 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790561432251112-7085-0/minicluster-data/master-0/wal/instance:
uuid: "309896d6be1144a184529fb8a1bc751b"
format_stamp: "Formatted at 2026-09-28 02:16:27 on dist-test-slave-86gv"
server_key: "b364768fa2220b872663a0a99924b147"
server_key_iv: "c32470bc1a09c13690c84687fc9e99f8"
server_key_version: "encryptionkey@0"
I20260928 02:16:27.160864 28827 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260928 02:16:27.164850 28842 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:16:27.166420 28827 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260928 02:16:27.166632 28827 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790561432251112-7085-0/minicluster-data/master-0/data,/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790561432251112-7085-0/minicluster-data/master-0/wal
uuid: "309896d6be1144a184529fb8a1bc751b"
format_stamp: "Formatted at 2026-09-28 02:16:27 on dist-test-slave-86gv"
server_key: "b364768fa2220b872663a0a99924b147"
server_key_iv: "c32470bc1a09c13690c84687fc9e99f8"
server_key_version: "encryptionkey@0"
I20260928 02:16:27.166890 28827 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790561432251112-7085-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790561432251112-7085-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790561432251112-7085-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:16:27.184445 28827 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:16:27.189338 28827 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:16:27.189640 28827 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:16:27.209357 28827 rpc_server.cc:307] RPC server started. Bound to: 127.6.235.126:39183
I20260928 02:16:27.209364 28893 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.235.126:39183 every 8 connection(s)
I20260928 02:16:27.210842 28827 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790561432251112-7085-0/minicluster-data/master-0/data/info.pb
I20260928 02:16:27.215070 28894 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:16:27.216068 7085 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTlaUuc/build/asan/bin/kudu as pid 28827
I20260928 02:16:27.216244 7085 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790561432251112-7085-0/minicluster-data/master-0/wal/instance
I20260928 02:16:27.216854 7085 external_mini_cluster.cc:1428] Setting key 994e5ca5880821ad0c498a83b30e9b6d
I20260928 02:16:27.230036 28894 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 309896d6be1144a184529fb8a1bc751b: Bootstrap starting.
Sep 28 02:16:27 dist-test-slave-86gv krb5kdc[28814](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561786, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.235.126@KRBTEST.COM
I20260928 02:16:27.233973 28894 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 309896d6be1144a184529fb8a1bc751b: Neither blocks nor log segments found. Creating new log.
I20260928 02:16:27.235117 28894 log.cc:824] T 00000000000000000000000000000000 P 309896d6be1144a184529fb8a1bc751b: Log is configured to *not* fsync() on all Append() calls
I20260928 02:16:27.238270 28894 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 309896d6be1144a184529fb8a1bc751b: No bootstrap required, opened a new log
I20260928 02:16:27.241233 28897 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:16:27.218721 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52493 (local address 127.6.235.126:39183)
0928 02:16:27.219510 (+ 789us) server_negotiation.cc:206] Beginning negotiation
0928 02:16:27.219522 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0928 02:16:27.219564 (+ 42us) server_negotiation.cc:407] Connection header received
0928 02:16:27.221636 (+ 2072us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:16:27.221660 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:16:27.222095 (+ 435us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:16:27.222435 (+ 340us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:16:27.224242 (+ 1807us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:16:27.225615 (+ 1373us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:16:27.227451 (+ 1836us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:16:27.228173 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:16:27.228319 (+ 146us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:16:27.233379 (+ 5060us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:16:27.233388 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:16:27.233406 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:16:27.233452 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:16:27.236701 (+ 3249us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:16:27.237621 (+ 920us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:16:27.237628 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:16:27.237634 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:16:27.237815 (+ 181us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:16:27.238303 (+ 488us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:16:27.238310 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:16:27.238318 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:16:27.238917 (+ 599us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:16:27.239264 (+ 347us) server_negotiation.cc:1035] Waiting for connection context
0928 02:16:27.239735 (+ 471us) server_negotiation.cc:299] Negotiation successful
0928 02:16:27.240079 (+ 344us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":561,"thread_start_us":356,"threads_started":1}
I20260928 02:16:27.245556 7085 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 02:16:27.247284 28894 raft_consensus.cc:348] T 00000000000000000000000000000000 P 309896d6be1144a184529fb8a1bc751b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "309896d6be1144a184529fb8a1bc751b" member_type: VOTER last_known_addr { host: "127.6.235.126" port: 39183 } }
I20260928 02:16:27.247700 28894 raft_consensus.cc:374] T 00000000000000000000000000000000 P 309896d6be1144a184529fb8a1bc751b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:16:27.247812 28894 raft_consensus.cc:729] T 00000000000000000000000000000000 P 309896d6be1144a184529fb8a1bc751b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 309896d6be1144a184529fb8a1bc751b, State: Initialized, Role: FOLLOWER
I20260928 02:16:27.248545 28894 consensus_queue.cc:260] T 00000000000000000000000000000000 P 309896d6be1144a184529fb8a1bc751b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "309896d6be1144a184529fb8a1bc751b" member_type: VOTER last_known_addr { host: "127.6.235.126" port: 39183 } }
I20260928 02:16:27.248792 28894 raft_consensus.cc:388] T 00000000000000000000000000000000 P 309896d6be1144a184529fb8a1bc751b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 02:16:27.248991 28894 raft_consensus.cc:482] T 00000000000000000000000000000000 P 309896d6be1144a184529fb8a1bc751b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 02:16:27.249191 28894 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 309896d6be1144a184529fb8a1bc751b [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:16:27.251988 28894 raft_consensus.cc:504] T 00000000000000000000000000000000 P 309896d6be1144a184529fb8a1bc751b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "309896d6be1144a184529fb8a1bc751b" member_type: VOTER last_known_addr { host: "127.6.235.126" port: 39183 } }
I20260928 02:16:27.252518 28894 leader_election.cc:302] T 00000000000000000000000000000000 P 309896d6be1144a184529fb8a1bc751b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 309896d6be1144a184529fb8a1bc751b; no voters:
I20260928 02:16:27.253269 28894 leader_election.cc:288] T 00000000000000000000000000000000 P 309896d6be1144a184529fb8a1bc751b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 02:16:27.253394 28904 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 309896d6be1144a184529fb8a1bc751b [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:16:27.254776 28904 raft_consensus.cc:686] T 00000000000000000000000000000000 P 309896d6be1144a184529fb8a1bc751b [term 1 LEADER]: Becoming Leader. State: Replica: 309896d6be1144a184529fb8a1bc751b, State: Running, Role: LEADER
I20260928 02:16:27.255455 28904 consensus_queue.cc:237] T 00000000000000000000000000000000 P 309896d6be1144a184529fb8a1bc751b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "309896d6be1144a184529fb8a1bc751b" member_type: VOTER last_known_addr { host: "127.6.235.126" port: 39183 } }
I20260928 02:16:27.255981 28894 sys_catalog.cc:564] T 00000000000000000000000000000000 P 309896d6be1144a184529fb8a1bc751b [sys.catalog]: configured and running, proceeding with master startup.
I20260928 02:16:27.258663 28905 sys_catalog.cc:455] T 00000000000000000000000000000000 P 309896d6be1144a184529fb8a1bc751b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 309896d6be1144a184529fb8a1bc751b. Latest consensus state: current_term: 1 leader_uuid: "309896d6be1144a184529fb8a1bc751b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "309896d6be1144a184529fb8a1bc751b" member_type: VOTER last_known_addr { host: "127.6.235.126" port: 39183 } } }
I20260928 02:16:27.259140 28905 sys_catalog.cc:458] T 00000000000000000000000000000000 P 309896d6be1144a184529fb8a1bc751b [sys.catalog]: This master's current role is: LEADER
I20260928 02:16:27.262938 28906 sys_catalog.cc:455] T 00000000000000000000000000000000 P 309896d6be1144a184529fb8a1bc751b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "309896d6be1144a184529fb8a1bc751b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "309896d6be1144a184529fb8a1bc751b" member_type: VOTER last_known_addr { host: "127.6.235.126" port: 39183 } } }
I20260928 02:16:27.263327 28906 sys_catalog.cc:458] T 00000000000000000000000000000000 P 309896d6be1144a184529fb8a1bc751b [sys.catalog]: This master's current role is: LEADER
I20260928 02:16:27.264810 28909 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 02:16:27.267377 28897 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:16:27.250136 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49043 (local address 127.6.235.126:39183)
0928 02:16:27.250395 (+ 259us) server_negotiation.cc:206] Beginning negotiation
0928 02:16:27.250403 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 02:16:27.250775 (+ 372us) server_negotiation.cc:407] Connection header received
0928 02:16:27.250985 (+ 210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:16:27.250990 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:16:27.251068 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:16:27.251183 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:16:27.252647 (+ 1464us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:16:27.253610 (+ 963us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:16:27.255257 (+ 1647us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:16:27.255916 (+ 659us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:16:27.256064 (+ 148us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:16:27.259301 (+ 3237us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:16:27.259310 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:16:27.259316 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:16:27.259361 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:16:27.263843 (+ 4482us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:16:27.264979 (+ 1136us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:16:27.264986 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:16:27.264992 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:16:27.265197 (+ 205us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:16:27.265764 (+ 567us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:16:27.265769 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:16:27.265776 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:16:27.266097 (+ 321us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:16:27.266375 (+ 278us) server_negotiation.cc:1035] Waiting for connection context
0928 02:16:27.266765 (+ 390us) server_negotiation.cc:299] Negotiation successful
0928 02:16:27.266963 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260928 02:16:27.270751 28909 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 02:16:27.281136 28909 catalog_manager.cc:1347] Generated new cluster ID: 00a577c86e9a4ad8bed2e7029d08a766
I20260928 02:16:27.281314 28909 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 02:16:27.304562 28909 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 02:16:27.306036 28909 catalog_manager.cc:1504] Loading token signing keys...
I20260928 02:16:27.316001 28909 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 309896d6be1144a184529fb8a1bc751b: Generated new TSK 0
I20260928 02:16:27.317057 28909 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 02:16:27.324050 7085 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskTlaUuc/build/asan/bin/kudu pid 28827
I20260928 02:16:27.325487 28889 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 02:16:27.418860 7085 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTlaUuc/build/asan/bin/kudu with pid 28827
W20260928 02:16:27.437269 28812 connection.cc:535] client connection to 127.6.235.126:39183 recv error: Network error: failed to read from TLS socket (remote: 127.6.235.126:39183): Connection reset by peer (error 104)
2026-09-28T02:16:27Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/12 (2140 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/13
Loading random data
Initializing database '/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790561432251112-7085-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 02:16:27 dist-test-slave-86gv krb5kdc[28926](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 02:16:27 dist-test-slave-86gv krb5kdc[28926](info): set up 2 sockets
Sep 28 02:16:27 dist-test-slave-86gv krb5kdc[28926](info): commencing operation
krb5kdc: starting...
W20260928 02:16:28.573789 7085 mini_kdc.cc:121] Time spent starting KDC: real 1.117s user 0.009s sys 0.022s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790561432251112-7085-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790561432251112-7085-0/krb5kdc/test-user.keytab.
Sep 28 02:16:28 dist-test-slave-86gv krb5kdc[28926](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561788, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T02:16:28Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T02:16:28Z Disabled control of system clock
WARNING: no policy specified for kudu/127.6.235.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.235.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.235.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790561432251112-7085-0/krb5kdc/kudu_127.6.235.126.keytab.
Entry for principal kudu/127.6.235.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790561432251112-7085-0/krb5kdc/kudu_127.6.235.126.keytab.
I20260928 02:16:28.862535 7085 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTlaUuc/build/asan/bin/kudu
/tmp/dist-test-taskTlaUuc/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790561432251112-7085-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790561432251112-7085-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790561432251112-7085-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790561432251112-7085-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.235.126:33405
--webserver_interface=127.6.235.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790561432251112-7085-0/krb5kdc/kudu_127.6.235.126.keytab
--principal=kudu/127.6.235.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.235.84:35597
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.235.126:33405
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790561432251112-7085-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790561432251112-7085-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790561432251112-7085-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:16:29.213457 28939 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:16:29.213784 28939 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:16:29.213980 28939 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:16:29.221786 28939 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 02:16:29.221930 28939 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:16:29.222025 28939 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:16:29.222124 28939 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 02:16:29.222211 28939 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 02:16:29.232568 28939 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.235.84:35597
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790561432251112-7085-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790561432251112-7085-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.235.126:33405
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790561432251112-7085-0/krb5kdc/kudu_127.6.235.126.keytab
--principal=kudu/127.6.235.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.235.126:33405
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790561432251112-7085-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.235.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790561432251112-7085-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 00:51:59 UTC on ccycloud.quasar-axgkwn.root.comops.site
build id 4676464
ASAN enabled
I20260928 02:16:29.234045 28939 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:16:29.235453 28939 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:16:29.248026 28944 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:16:29.248064 28947 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:16:29.247991 28945 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:16:29.249347 28939 server_base.cc:1034] running on GCE node
I20260928 02:16:29.250187 28939 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:16:29.251730 28939 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:16:29.253000 28939 hybrid_clock.cc:648] HybridClock initialized: now 1790561789252935 us; error 56 us; skew 500 ppm
I20260928 02:16:29.253389 28939 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 02:16:29 dist-test-slave-86gv krb5kdc[28926](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561789, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.235.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:16:29.256884 28939 init.cc:377] Logged in from keytab as kudu/127.6.235.126@KRBTEST.COM (short username kudu)
I20260928 02:16:29.258544 28939 webserver.cc:460] Webserver started at http://127.6.235.126:37289/ using document root <none> and password file <none>
I20260928 02:16:29.259280 28939 fs_manager.cc:359] Metadata directory not provided
I20260928 02:16:29.259413 28939 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:16:29.259742 28939 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:16:29.262918 28939 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790561432251112-7085-0/minicluster-data/master-0/data/instance:
uuid: "9855a0e147394f5195a6ec2f6b4e303c"
format_stamp: "Formatted at 2026-09-28 02:16:29 on dist-test-slave-86gv"
server_key: "1dcc3ab989e0f5d0fb83b02ffa809595"
server_key_iv: "4b379ddfdde1273f80f70ab9f1df83f2"
server_key_version: "encryptionkey@0"
I20260928 02:16:29.263698 28939 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790561432251112-7085-0/minicluster-data/master-0/wal/instance:
uuid: "9855a0e147394f5195a6ec2f6b4e303c"
format_stamp: "Formatted at 2026-09-28 02:16:29 on dist-test-slave-86gv"
server_key: "1dcc3ab989e0f5d0fb83b02ffa809595"
server_key_iv: "4b379ddfdde1273f80f70ab9f1df83f2"
server_key_version: "encryptionkey@0"
I20260928 02:16:29.269299 28939 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20260928 02:16:29.273108 28954 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:16:29.274590 28939 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260928 02:16:29.274792 28939 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790561432251112-7085-0/minicluster-data/master-0/data,/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790561432251112-7085-0/minicluster-data/master-0/wal
uuid: "9855a0e147394f5195a6ec2f6b4e303c"
format_stamp: "Formatted at 2026-09-28 02:16:29 on dist-test-slave-86gv"
server_key: "1dcc3ab989e0f5d0fb83b02ffa809595"
server_key_iv: "4b379ddfdde1273f80f70ab9f1df83f2"
server_key_version: "encryptionkey@0"
I20260928 02:16:29.275010 28939 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790561432251112-7085-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790561432251112-7085-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790561432251112-7085-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:16:29.293783 28939 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:16:29.298264 28939 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:16:29.298630 28939 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:16:29.318449 28939 rpc_server.cc:307] RPC server started. Bound to: 127.6.235.126:33405
I20260928 02:16:29.318461 29005 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.235.126:33405 every 8 connection(s)
I20260928 02:16:29.319933 28939 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790561432251112-7085-0/minicluster-data/master-0/data/info.pb
I20260928 02:16:29.324008 29006 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:16:29.329389 7085 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTlaUuc/build/asan/bin/kudu as pid 28939
I20260928 02:16:29.329600 7085 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790561432251112-7085-0/minicluster-data/master-0/wal/instance
I20260928 02:16:29.330083 7085 external_mini_cluster.cc:1428] Setting key 37e61093a3cadffad1a99a05d0aabfbf
I20260928 02:16:29.335481 29006 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9855a0e147394f5195a6ec2f6b4e303c: Bootstrap starting.
I20260928 02:16:29.338852 29006 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9855a0e147394f5195a6ec2f6b4e303c: Neither blocks nor log segments found. Creating new log.
I20260928 02:16:29.339999 29006 log.cc:824] T 00000000000000000000000000000000 P 9855a0e147394f5195a6ec2f6b4e303c: Log is configured to *not* fsync() on all Append() calls
I20260928 02:16:29.343092 29006 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9855a0e147394f5195a6ec2f6b4e303c: No bootstrap required, opened a new log
I20260928 02:16:29.343828 29009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:16:29.332166 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52077 (local address 127.6.235.126:33405)
0928 02:16:29.333100 (+ 934us) server_negotiation.cc:206] Beginning negotiation
0928 02:16:29.333112 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0928 02:16:29.333158 (+ 46us) server_negotiation.cc:407] Connection header received
0928 02:16:29.334883 (+ 1725us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:16:29.334907 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:16:29.335340 (+ 433us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:16:29.335746 (+ 406us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:16:29.336374 (+ 628us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:16:29.337806 (+ 1432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:16:29.339236 (+ 1430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:16:29.339948 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:16:29.340163 (+ 215us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:16:29.341467 (+ 1304us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:16:29.341475 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:16:29.341491 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 02:16:29.342043 (+ 552us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:16:29.342120 (+ 77us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 02:16:29.342144 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:16:29.342326 (+ 182us) server_negotiation.cc:1035] Waiting for connection context
0928 02:16:29.342597 (+ 271us) server_negotiation.cc:299] Negotiation successful
0928 02:16:29.342766 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":665,"thread_start_us":414,"threads_started":1}
I20260928 02:16:29.347827 7085 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 02:16:29.351547 29006 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9855a0e147394f5195a6ec2f6b4e303c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9855a0e147394f5195a6ec2f6b4e303c" member_type: VOTER last_known_addr { host: "127.6.235.126" port: 33405 } }
I20260928 02:16:29.351965 29006 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9855a0e147394f5195a6ec2f6b4e303c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:16:29.352099 29006 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9855a0e147394f5195a6ec2f6b4e303c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9855a0e147394f5195a6ec2f6b4e303c, State: Initialized, Role: FOLLOWER
I20260928 02:16:29.352787 29006 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9855a0e147394f5195a6ec2f6b4e303c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9855a0e147394f5195a6ec2f6b4e303c" member_type: VOTER last_known_addr { host: "127.6.235.126" port: 33405 } }
I20260928 02:16:29.353097 29006 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9855a0e147394f5195a6ec2f6b4e303c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 02:16:29.353267 29006 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9855a0e147394f5195a6ec2f6b4e303c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 02:16:29.353478 29006 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9855a0e147394f5195a6ec2f6b4e303c [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:16:29.356604 29006 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9855a0e147394f5195a6ec2f6b4e303c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9855a0e147394f5195a6ec2f6b4e303c" member_type: VOTER last_known_addr { host: "127.6.235.126" port: 33405 } }
I20260928 02:16:29.357321 29006 leader_election.cc:302] T 00000000000000000000000000000000 P 9855a0e147394f5195a6ec2f6b4e303c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9855a0e147394f5195a6ec2f6b4e303c; no voters:
I20260928 02:16:29.358021 29006 leader_election.cc:288] T 00000000000000000000000000000000 P 9855a0e147394f5195a6ec2f6b4e303c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 02:16:29.358163 29016 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9855a0e147394f5195a6ec2f6b4e303c [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:16:29.359869 29016 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9855a0e147394f5195a6ec2f6b4e303c [term 1 LEADER]: Becoming Leader. State: Replica: 9855a0e147394f5195a6ec2f6b4e303c, State: Running, Role: LEADER
I20260928 02:16:29.360533 29009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:16:29.352468 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57045 (local address 127.6.235.126:33405)
0928 02:16:29.352693 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0928 02:16:29.352700 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 02:16:29.353392 (+ 692us) server_negotiation.cc:407] Connection header received
0928 02:16:29.353613 (+ 221us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:16:29.353618 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:16:29.353707 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:16:29.353849 (+ 142us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:16:29.354299 (+ 450us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:16:29.355480 (+ 1181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:16:29.356820 (+ 1340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:16:29.357508 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:16:29.357638 (+ 130us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:16:29.359148 (+ 1510us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:16:29.359154 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:16:29.359159 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 02:16:29.359186 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:16:29.359266 (+ 80us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 02:16:29.359298 (+ 32us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:16:29.359423 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0928 02:16:29.360081 (+ 658us) server_negotiation.cc:299] Negotiation successful
0928 02:16:29.360144 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260928 02:16:29.360527 29016 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9855a0e147394f5195a6ec2f6b4e303c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9855a0e147394f5195a6ec2f6b4e303c" member_type: VOTER last_known_addr { host: "127.6.235.126" port: 33405 } }
I20260928 02:16:29.361274 29006 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9855a0e147394f5195a6ec2f6b4e303c [sys.catalog]: configured and running, proceeding with master startup.
I20260928 02:16:29.363721 29018 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9855a0e147394f5195a6ec2f6b4e303c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9855a0e147394f5195a6ec2f6b4e303c. Latest consensus state: current_term: 1 leader_uuid: "9855a0e147394f5195a6ec2f6b4e303c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9855a0e147394f5195a6ec2f6b4e303c" member_type: VOTER last_known_addr { host: "127.6.235.126" port: 33405 } } }
I20260928 02:16:29.364122 29018 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9855a0e147394f5195a6ec2f6b4e303c [sys.catalog]: This master's current role is: LEADER
I20260928 02:16:29.363845 29017 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9855a0e147394f5195a6ec2f6b4e303c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9855a0e147394f5195a6ec2f6b4e303c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9855a0e147394f5195a6ec2f6b4e303c" member_type: VOTER last_known_addr { host: "127.6.235.126" port: 33405 } } }
I20260928 02:16:29.364254 29017 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9855a0e147394f5195a6ec2f6b4e303c [sys.catalog]: This master's current role is: LEADER
I20260928 02:16:29.370436 29021 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 02:16:29.375452 29021 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 02:16:29.385413 29021 catalog_manager.cc:1347] Generated new cluster ID: 0994ec8688494524bbcd0023119e9062
I20260928 02:16:29.385550 29021 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 02:16:29.415501 29021 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 02:16:29.416882 29021 catalog_manager.cc:1504] Loading token signing keys...
I20260928 02:16:29.428689 29021 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9855a0e147394f5195a6ec2f6b4e303c: Generated new TSK 0
I20260928 02:16:29.429622 29021 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 02:16:29.440173 7085 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskTlaUuc/build/asan/bin/kudu pid 28939
W20260928 02:16:29.441562 29001 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:52077
W20260928 02:16:29.442461 7085 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskTlaUuc/build/asan/bin/kudu pid 28939: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260928 02:16:29.442653 7085 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTlaUuc/build/asan/bin/kudu with pid 28939
W20260928 02:16:29.460717 28924 connection.cc:535] client connection to 127.6.235.126:33405 recv error: Network error: failed to read from TLS socket (remote: 127.6.235.126:33405): Connection reset by peer (error 104)
2026-09-28T02:16:29Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/13 (2023 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/14
Loading random data
Initializing database '/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790561432251112-7085-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 02:16:29 dist-test-slave-86gv krb5kdc[29037](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 02:16:29 dist-test-slave-86gv krb5kdc[29037](info): set up 2 sockets
Sep 28 02:16:29 dist-test-slave-86gv krb5kdc[29037](info): commencing operation
krb5kdc: starting...
W20260928 02:16:30.585807 7085 mini_kdc.cc:121] Time spent starting KDC: real 1.106s user 0.000s sys 0.029s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790561432251112-7085-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790561432251112-7085-0/krb5kdc/test-user.keytab.
Sep 28 02:16:30 dist-test-slave-86gv krb5kdc[29037](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561790, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T02:16:30Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T02:16:30Z Disabled control of system clock
WARNING: no policy specified for kudu/127.6.235.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.235.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.235.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790561432251112-7085-0/krb5kdc/kudu_127.6.235.126.keytab.
Entry for principal kudu/127.6.235.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790561432251112-7085-0/krb5kdc/kudu_127.6.235.126.keytab.
I20260928 02:16:30.865871 7085 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTlaUuc/build/asan/bin/kudu
/tmp/dist-test-taskTlaUuc/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790561432251112-7085-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790561432251112-7085-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790561432251112-7085-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790561432251112-7085-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.235.126:38741
--webserver_interface=127.6.235.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790561432251112-7085-0/krb5kdc/kudu_127.6.235.126.keytab
--principal=kudu/127.6.235.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.235.84:41241
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.235.126:38741
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790561432251112-7085-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790561432251112-7085-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790561432251112-7085-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:16:31.218631 29050 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:16:31.218964 29050 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:16:31.219134 29050 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:16:31.229580 29050 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 02:16:31.229760 29050 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:16:31.229839 29050 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:16:31.229928 29050 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 02:16:31.229995 29050 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 02:16:31.244221 29050 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.235.84:41241
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790561432251112-7085-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790561432251112-7085-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.235.126:38741
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790561432251112-7085-0/krb5kdc/kudu_127.6.235.126.keytab
--principal=kudu/127.6.235.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.235.126:38741
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790561432251112-7085-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.235.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790561432251112-7085-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 00:51:59 UTC on ccycloud.quasar-axgkwn.root.comops.site
build id 4676464
ASAN enabled
I20260928 02:16:31.245817 29050 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:16:31.247215 29050 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:16:31.259613 29056 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:16:31.259792 29058 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:16:31.259822 29055 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:16:31.261289 29050 server_base.cc:1034] running on GCE node
I20260928 02:16:31.262183 29050 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:16:31.263846 29050 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:16:31.265089 29050 hybrid_clock.cc:648] HybridClock initialized: now 1790561791265040 us; error 40 us; skew 500 ppm
I20260928 02:16:31.265498 29050 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 02:16:31 dist-test-slave-86gv krb5kdc[29037](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561791, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.235.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:16:31.268874 29050 init.cc:377] Logged in from keytab as kudu/127.6.235.126@KRBTEST.COM (short username kudu)
I20260928 02:16:31.270475 29050 webserver.cc:460] Webserver started at http://127.6.235.126:42529/ using document root <none> and password file <none>
I20260928 02:16:31.271217 29050 fs_manager.cc:359] Metadata directory not provided
I20260928 02:16:31.271363 29050 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:16:31.271672 29050 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:16:31.274915 29050 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790561432251112-7085-0/minicluster-data/master-0/data/instance:
uuid: "6b4332715ef9445a924bef377f214239"
format_stamp: "Formatted at 2026-09-28 02:16:31 on dist-test-slave-86gv"
server_key: "fde185e5ef4da04097548e793c5321a9"
server_key_iv: "457f66e4daf11d08d1ab4e167cf67c36"
server_key_version: "encryptionkey@0"
I20260928 02:16:31.275714 29050 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790561432251112-7085-0/minicluster-data/master-0/wal/instance:
uuid: "6b4332715ef9445a924bef377f214239"
format_stamp: "Formatted at 2026-09-28 02:16:31 on dist-test-slave-86gv"
server_key: "fde185e5ef4da04097548e793c5321a9"
server_key_iv: "457f66e4daf11d08d1ab4e167cf67c36"
server_key_version: "encryptionkey@0"
I20260928 02:16:31.281123 29050 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.005s
I20260928 02:16:31.284667 29065 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:16:31.286156 29050 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 02:16:31.286346 29050 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790561432251112-7085-0/minicluster-data/master-0/data,/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790561432251112-7085-0/minicluster-data/master-0/wal
uuid: "6b4332715ef9445a924bef377f214239"
format_stamp: "Formatted at 2026-09-28 02:16:31 on dist-test-slave-86gv"
server_key: "fde185e5ef4da04097548e793c5321a9"
server_key_iv: "457f66e4daf11d08d1ab4e167cf67c36"
server_key_version: "encryptionkey@0"
I20260928 02:16:31.286556 29050 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790561432251112-7085-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790561432251112-7085-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790561432251112-7085-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:16:31.292452 29050 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:16:31.296622 29050 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:16:31.297016 29050 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:16:31.316571 29050 rpc_server.cc:307] RPC server started. Bound to: 127.6.235.126:38741
I20260928 02:16:31.316578 29116 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.235.126:38741 every 8 connection(s)
I20260928 02:16:31.318094 29050 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790561432251112-7085-0/minicluster-data/master-0/data/info.pb
I20260928 02:16:31.322213 7085 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTlaUuc/build/asan/bin/kudu as pid 29050
I20260928 02:16:31.322391 7085 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790561432251112-7085-0/minicluster-data/master-0/wal/instance
I20260928 02:16:31.322641 29117 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:16:31.322892 7085 external_mini_cluster.cc:1428] Setting key d7cbafcfc5678a6abd7ea45316790b83
I20260928 02:16:31.332543 29120 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:16:31.324813 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59173 (local address 127.6.235.126:38741)
0928 02:16:31.325809 (+ 996us) server_negotiation.cc:206] Beginning negotiation
0928 02:16:31.325826 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0928 02:16:31.325885 (+ 59us) server_negotiation.cc:407] Connection header received
0928 02:16:31.328163 (+ 2278us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:16:31.328196 (+ 33us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:16:31.328723 (+ 527us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:16:31.329245 (+ 522us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:16:31.330372 (+ 1127us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:16:31.330388 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:16:31.330406 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 02:16:31.330439 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:16:31.330514 (+ 75us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 02:16:31.330561 (+ 47us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:16:31.330810 (+ 249us) server_negotiation.cc:1035] Waiting for connection context
0928 02:16:31.330976 (+ 166us) server_negotiation.cc:299] Negotiation successful
0928 02:16:31.331190 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":669,"thread_start_us":366,"threads_started":1}
I20260928 02:16:31.336607 7085 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 02:16:31.338775 29117 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6b4332715ef9445a924bef377f214239: Bootstrap starting.
I20260928 02:16:31.342636 29117 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6b4332715ef9445a924bef377f214239: Neither blocks nor log segments found. Creating new log.
I20260928 02:16:31.343621 29120 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:16:31.341112 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54489 (local address 127.6.235.126:38741)
0928 02:16:31.341361 (+ 249us) server_negotiation.cc:206] Beginning negotiation
0928 02:16:31.341369 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 02:16:31.341690 (+ 321us) server_negotiation.cc:407] Connection header received
0928 02:16:31.341890 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:16:31.341900 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:16:31.341948 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:16:31.342045 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:16:31.342972 (+ 927us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:16:31.342977 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:16:31.342982 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 02:16:31.343003 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:16:31.343050 (+ 47us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 02:16:31.343065 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:16:31.343174 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0928 02:16:31.343285 (+ 111us) server_negotiation.cc:299] Negotiation successful
0928 02:16:31.343345 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260928 02:16:31.344023 29117 log.cc:824] T 00000000000000000000000000000000 P 6b4332715ef9445a924bef377f214239: Log is configured to *not* fsync() on all Append() calls
I20260928 02:16:31.347180 29117 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6b4332715ef9445a924bef377f214239: No bootstrap required, opened a new log
I20260928 02:16:31.355966 29117 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6b4332715ef9445a924bef377f214239 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6b4332715ef9445a924bef377f214239" member_type: VOTER last_known_addr { host: "127.6.235.126" port: 38741 } }
I20260928 02:16:31.356438 29117 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6b4332715ef9445a924bef377f214239 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:16:31.356583 29117 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6b4332715ef9445a924bef377f214239 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6b4332715ef9445a924bef377f214239, State: Initialized, Role: FOLLOWER
I20260928 02:16:31.357366 29117 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6b4332715ef9445a924bef377f214239 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6b4332715ef9445a924bef377f214239" member_type: VOTER last_known_addr { host: "127.6.235.126" port: 38741 } }
I20260928 02:16:31.357616 29117 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6b4332715ef9445a924bef377f214239 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 02:16:31.357788 29117 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6b4332715ef9445a924bef377f214239 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 02:16:31.357972 29117 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6b4332715ef9445a924bef377f214239 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:16:31.360397 29117 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6b4332715ef9445a924bef377f214239 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6b4332715ef9445a924bef377f214239" member_type: VOTER last_known_addr { host: "127.6.235.126" port: 38741 } }
I20260928 02:16:31.361076 29117 leader_election.cc:302] T 00000000000000000000000000000000 P 6b4332715ef9445a924bef377f214239 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6b4332715ef9445a924bef377f214239; no voters:
I20260928 02:16:31.361716 29117 leader_election.cc:288] T 00000000000000000000000000000000 P 6b4332715ef9445a924bef377f214239 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 02:16:31.361876 29127 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6b4332715ef9445a924bef377f214239 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:16:31.363365 29127 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6b4332715ef9445a924bef377f214239 [term 1 LEADER]: Becoming Leader. State: Replica: 6b4332715ef9445a924bef377f214239, State: Running, Role: LEADER
I20260928 02:16:31.363984 29127 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6b4332715ef9445a924bef377f214239 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6b4332715ef9445a924bef377f214239" member_type: VOTER last_known_addr { host: "127.6.235.126" port: 38741 } }
I20260928 02:16:31.364132 29117 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6b4332715ef9445a924bef377f214239 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 02:16:31.366943 29129 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6b4332715ef9445a924bef377f214239 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6b4332715ef9445a924bef377f214239. Latest consensus state: current_term: 1 leader_uuid: "6b4332715ef9445a924bef377f214239" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6b4332715ef9445a924bef377f214239" member_type: VOTER last_known_addr { host: "127.6.235.126" port: 38741 } } }
I20260928 02:16:31.367443 29129 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6b4332715ef9445a924bef377f214239 [sys.catalog]: This master's current role is: LEADER
I20260928 02:16:31.367403 29128 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6b4332715ef9445a924bef377f214239 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6b4332715ef9445a924bef377f214239" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6b4332715ef9445a924bef377f214239" member_type: VOTER last_known_addr { host: "127.6.235.126" port: 38741 } } }
I20260928 02:16:31.367892 29128 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6b4332715ef9445a924bef377f214239 [sys.catalog]: This master's current role is: LEADER
I20260928 02:16:31.369170 29136 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 02:16:31.375517 29136 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 02:16:31.385808 29136 catalog_manager.cc:1347] Generated new cluster ID: cf35b24bfc6b44e7ac486c16679a4cbc
I20260928 02:16:31.385974 29136 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 02:16:31.402395 29136 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 02:16:31.402894 29136 catalog_manager.cc:1504] Loading token signing keys...
I20260928 02:16:31.410195 29136 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6b4332715ef9445a924bef377f214239: Generated new TSK 0
I20260928 02:16:31.411155 29136 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 02:16:31.415462 7085 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskTlaUuc/build/asan/bin/kudu pid 29050
W20260928 02:16:31.416781 29112 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:59173
W20260928 02:16:31.417598 7085 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskTlaUuc/build/asan/bin/kudu pid 29050: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260928 02:16:31.417753 7085 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTlaUuc/build/asan/bin/kudu with pid 29050
2026-09-28T02:16:31Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/14 (1971 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/15
Loading random data
Initializing database '/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790561432251112-7085-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 02:16:31 dist-test-slave-86gv krb5kdc[29148](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 02:16:31 dist-test-slave-86gv krb5kdc[29148](info): set up 2 sockets
Sep 28 02:16:31 dist-test-slave-86gv krb5kdc[29148](info): commencing operation
krb5kdc: starting...
W20260928 02:16:32.525849 7085 mini_kdc.cc:121] Time spent starting KDC: real 1.074s user 0.000s sys 0.029s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790561432251112-7085-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790561432251112-7085-0/krb5kdc/test-user.keytab.
Sep 28 02:16:32 dist-test-slave-86gv krb5kdc[29148](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561792, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T02:16:32Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T02:16:32Z Disabled control of system clock
WARNING: no policy specified for kudu/127.6.235.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.235.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.235.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790561432251112-7085-0/krb5kdc/kudu_127.6.235.126.keytab.
Entry for principal kudu/127.6.235.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790561432251112-7085-0/krb5kdc/kudu_127.6.235.126.keytab.
I20260928 02:16:32.809317 7085 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTlaUuc/build/asan/bin/kudu
/tmp/dist-test-taskTlaUuc/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790561432251112-7085-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790561432251112-7085-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790561432251112-7085-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790561432251112-7085-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.235.126:45777
--webserver_interface=127.6.235.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790561432251112-7085-0/krb5kdc/kudu_127.6.235.126.keytab
--principal=kudu/127.6.235.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.235.84:42737
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.235.126:45777
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790561432251112-7085-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790561432251112-7085-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790561432251112-7085-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:16:33.154827 29161 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:16:33.155176 29161 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:16:33.155337 29161 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:16:33.162951 29161 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 02:16:33.163096 29161 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:16:33.163193 29161 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:16:33.163296 29161 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 02:16:33.163380 29161 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 02:16:33.173652 29161 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.235.84:42737
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790561432251112-7085-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790561432251112-7085-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.235.126:45777
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790561432251112-7085-0/krb5kdc/kudu_127.6.235.126.keytab
--principal=kudu/127.6.235.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.235.126:45777
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790561432251112-7085-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.235.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790561432251112-7085-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 00:51:59 UTC on ccycloud.quasar-axgkwn.root.comops.site
build id 4676464
ASAN enabled
I20260928 02:16:33.175050 29161 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:16:33.176379 29161 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:16:33.188372 29167 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:16:33.188319 29169 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:16:33.189178 29166 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:16:33.189322 29161 server_base.cc:1034] running on GCE node
I20260928 02:16:33.190124 29161 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:16:33.191633 29161 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:16:33.192883 29161 hybrid_clock.cc:648] HybridClock initialized: now 1790561793192819 us; error 56 us; skew 500 ppm
I20260928 02:16:33.193336 29161 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 02:16:33 dist-test-slave-86gv krb5kdc[29148](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561793, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.235.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:16:33.196853 29161 init.cc:377] Logged in from keytab as kudu/127.6.235.126@KRBTEST.COM (short username kudu)
I20260928 02:16:33.198413 29161 webserver.cc:460] Webserver started at http://127.6.235.126:35315/ using document root <none> and password file <none>
I20260928 02:16:33.199146 29161 fs_manager.cc:359] Metadata directory not provided
I20260928 02:16:33.199281 29161 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:16:33.199608 29161 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:16:33.202791 29161 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790561432251112-7085-0/minicluster-data/master-0/data/instance:
uuid: "ab9c89d727134b53b0f4b2cb856e0364"
format_stamp: "Formatted at 2026-09-28 02:16:33 on dist-test-slave-86gv"
server_key: "8fbf479f8b6d35970ee675f1fe10a6e4"
server_key_iv: "94ce80cdec8bdfc691c064f9b0fcebb2"
server_key_version: "encryptionkey@0"
I20260928 02:16:33.203615 29161 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790561432251112-7085-0/minicluster-data/master-0/wal/instance:
uuid: "ab9c89d727134b53b0f4b2cb856e0364"
format_stamp: "Formatted at 2026-09-28 02:16:33 on dist-test-slave-86gv"
server_key: "8fbf479f8b6d35970ee675f1fe10a6e4"
server_key_iv: "94ce80cdec8bdfc691c064f9b0fcebb2"
server_key_version: "encryptionkey@0"
I20260928 02:16:33.209229 29161 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260928 02:16:33.212837 29176 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:16:33.214291 29161 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260928 02:16:33.214493 29161 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790561432251112-7085-0/minicluster-data/master-0/data,/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790561432251112-7085-0/minicluster-data/master-0/wal
uuid: "ab9c89d727134b53b0f4b2cb856e0364"
format_stamp: "Formatted at 2026-09-28 02:16:33 on dist-test-slave-86gv"
server_key: "8fbf479f8b6d35970ee675f1fe10a6e4"
server_key_iv: "94ce80cdec8bdfc691c064f9b0fcebb2"
server_key_version: "encryptionkey@0"
I20260928 02:16:33.214706 29161 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790561432251112-7085-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790561432251112-7085-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790561432251112-7085-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:16:33.235333 29161 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:16:33.240008 29161 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:16:33.240269 29161 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:16:33.259078 29161 rpc_server.cc:307] RPC server started. Bound to: 127.6.235.126:45777
I20260928 02:16:33.259099 29227 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.235.126:45777 every 8 connection(s)
I20260928 02:16:33.260743 29161 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790561432251112-7085-0/minicluster-data/master-0/data/info.pb
I20260928 02:16:33.264988 29228 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:16:33.266341 7085 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTlaUuc/build/asan/bin/kudu as pid 29161
I20260928 02:16:33.266520 7085 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790561432251112-7085-0/minicluster-data/master-0/wal/instance
I20260928 02:16:33.266981 7085 external_mini_cluster.cc:1428] Setting key a5956db5a1471fbd24cc5fdbd43a8cce
I20260928 02:16:33.279971 29228 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ab9c89d727134b53b0f4b2cb856e0364: Bootstrap starting.
Sep 28 02:16:33 dist-test-slave-86gv krb5kdc[29148](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561792, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.235.126@KRBTEST.COM
I20260928 02:16:33.283315 29228 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ab9c89d727134b53b0f4b2cb856e0364: Neither blocks nor log segments found. Creating new log.
I20260928 02:16:33.284837 29228 log.cc:824] T 00000000000000000000000000000000 P ab9c89d727134b53b0f4b2cb856e0364: Log is configured to *not* fsync() on all Append() calls
I20260928 02:16:33.288357 29228 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ab9c89d727134b53b0f4b2cb856e0364: No bootstrap required, opened a new log
I20260928 02:16:33.291841 29231 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:16:33.268870 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51363 (local address 127.6.235.126:45777)
0928 02:16:33.269774 (+ 904us) server_negotiation.cc:206] Beginning negotiation
0928 02:16:33.269790 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0928 02:16:33.269844 (+ 54us) server_negotiation.cc:407] Connection header received
0928 02:16:33.271792 (+ 1948us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:16:33.271825 (+ 33us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:16:33.272432 (+ 607us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:16:33.272936 (+ 504us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:16:33.274343 (+ 1407us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:16:33.276370 (+ 2027us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:16:33.277738 (+ 1368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:16:33.278904 (+ 1166us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:16:33.279088 (+ 184us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:16:33.283195 (+ 4107us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:16:33.283209 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:16:33.283230 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:16:33.283301 (+ 71us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:16:33.287197 (+ 3896us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:16:33.287883 (+ 686us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:16:33.287891 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:16:33.287900 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:16:33.288105 (+ 205us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:16:33.288480 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:16:33.288484 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:16:33.288488 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:16:33.289278 (+ 790us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:16:33.289566 (+ 288us) server_negotiation.cc:1035] Waiting for connection context
0928 02:16:33.289912 (+ 346us) server_negotiation.cc:299] Negotiation successful
0928 02:16:33.290394 (+ 482us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":586,"thread_start_us":348,"threads_started":1}
I20260928 02:16:33.295713 7085 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 02:16:33.297910 29228 raft_consensus.cc:348] T 00000000000000000000000000000000 P ab9c89d727134b53b0f4b2cb856e0364 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ab9c89d727134b53b0f4b2cb856e0364" member_type: VOTER last_known_addr { host: "127.6.235.126" port: 45777 } }
I20260928 02:16:33.298365 29228 raft_consensus.cc:374] T 00000000000000000000000000000000 P ab9c89d727134b53b0f4b2cb856e0364 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:16:33.298489 29228 raft_consensus.cc:729] T 00000000000000000000000000000000 P ab9c89d727134b53b0f4b2cb856e0364 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ab9c89d727134b53b0f4b2cb856e0364, State: Initialized, Role: FOLLOWER
I20260928 02:16:33.299543 29228 consensus_queue.cc:260] T 00000000000000000000000000000000 P ab9c89d727134b53b0f4b2cb856e0364 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ab9c89d727134b53b0f4b2cb856e0364" member_type: VOTER last_known_addr { host: "127.6.235.126" port: 45777 } }
I20260928 02:16:33.299871 29236 raft_consensus.cc:482] T 00000000000000000000000000000000 P ab9c89d727134b53b0f4b2cb856e0364 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 02:16:33.300112 29236 raft_consensus.cc:504] T 00000000000000000000000000000000 P ab9c89d727134b53b0f4b2cb856e0364 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ab9c89d727134b53b0f4b2cb856e0364" member_type: VOTER last_known_addr { host: "127.6.235.126" port: 45777 } }
I20260928 02:16:33.300845 29236 leader_election.cc:302] T 00000000000000000000000000000000 P ab9c89d727134b53b0f4b2cb856e0364 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ab9c89d727134b53b0f4b2cb856e0364; no voters:
I20260928 02:16:33.299809 29228 raft_consensus.cc:388] T 00000000000000000000000000000000 P ab9c89d727134b53b0f4b2cb856e0364 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 02:16:33.301693 29236 leader_election.cc:288] T 00000000000000000000000000000000 P ab9c89d727134b53b0f4b2cb856e0364 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers
I20260928 02:16:33.301749 29228 raft_consensus.cc:482] T 00000000000000000000000000000000 P ab9c89d727134b53b0f4b2cb856e0364 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 02:16:33.301923 29228 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ab9c89d727134b53b0f4b2cb856e0364 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:16:33.305162 29228 raft_consensus.cc:504] T 00000000000000000000000000000000 P ab9c89d727134b53b0f4b2cb856e0364 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ab9c89d727134b53b0f4b2cb856e0364" member_type: VOTER last_known_addr { host: "127.6.235.126" port: 45777 } }
I20260928 02:16:33.305424 29228 leader_election.cc:302] T 00000000000000000000000000000000 P ab9c89d727134b53b0f4b2cb856e0364 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ab9c89d727134b53b0f4b2cb856e0364; no voters:
I20260928 02:16:33.305480 29239 raft_consensus.cc:2741] T 00000000000000000000000000000000 P ab9c89d727134b53b0f4b2cb856e0364 [term 1 FOLLOWER]: Leader pre-election decision vote started in defunct term 0: won
I20260928 02:16:33.305662 29228 leader_election.cc:288] T 00000000000000000000000000000000 P ab9c89d727134b53b0f4b2cb856e0364 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 02:16:33.306324 29240 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ab9c89d727134b53b0f4b2cb856e0364 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:16:33.306639 29240 raft_consensus.cc:686] T 00000000000000000000000000000000 P ab9c89d727134b53b0f4b2cb856e0364 [term 1 LEADER]: Becoming Leader. State: Replica: ab9c89d727134b53b0f4b2cb856e0364, State: Running, Role: LEADER
I20260928 02:16:33.307222 29240 consensus_queue.cc:237] T 00000000000000000000000000000000 P ab9c89d727134b53b0f4b2cb856e0364 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ab9c89d727134b53b0f4b2cb856e0364" member_type: VOTER last_known_addr { host: "127.6.235.126" port: 45777 } }
I20260928 02:16:33.307807 29228 sys_catalog.cc:564] T 00000000000000000000000000000000 P ab9c89d727134b53b0f4b2cb856e0364 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 02:16:33.309671 29236 sys_catalog.cc:455] T 00000000000000000000000000000000 P ab9c89d727134b53b0f4b2cb856e0364 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ab9c89d727134b53b0f4b2cb856e0364. Latest consensus state: current_term: 1 leader_uuid: "ab9c89d727134b53b0f4b2cb856e0364" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ab9c89d727134b53b0f4b2cb856e0364" member_type: VOTER last_known_addr { host: "127.6.235.126" port: 45777 } } }
I20260928 02:16:33.309617 29239 sys_catalog.cc:455] T 00000000000000000000000000000000 P ab9c89d727134b53b0f4b2cb856e0364 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ab9c89d727134b53b0f4b2cb856e0364" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ab9c89d727134b53b0f4b2cb856e0364" member_type: VOTER last_known_addr { host: "127.6.235.126" port: 45777 } } }
I20260928 02:16:33.309909 29236 sys_catalog.cc:458] T 00000000000000000000000000000000 P ab9c89d727134b53b0f4b2cb856e0364 [sys.catalog]: This master's current role is: LEADER
I20260928 02:16:33.309949 29239 sys_catalog.cc:458] T 00000000000000000000000000000000 P ab9c89d727134b53b0f4b2cb856e0364 [sys.catalog]: This master's current role is: LEADER
I20260928 02:16:33.321060 29231 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:16:33.300460 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56583 (local address 127.6.235.126:45777)
0928 02:16:33.300709 (+ 249us) server_negotiation.cc:206] Beginning negotiation
0928 02:16:33.300719 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 02:16:33.302320 (+ 1601us) server_negotiation.cc:407] Connection header received
0928 02:16:33.302529 (+ 209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:16:33.302537 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:16:33.302635 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:16:33.302734 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:16:33.304482 (+ 1748us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:16:33.305236 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:16:33.307075 (+ 1839us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:16:33.307777 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:16:33.307909 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:16:33.311612 (+ 3703us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:16:33.311623 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:16:33.311631 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:16:33.311682 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:16:33.316419 (+ 4737us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:16:33.317616 (+ 1197us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:16:33.317626 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:16:33.317633 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:16:33.317848 (+ 215us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:16:33.318414 (+ 566us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:16:33.318423 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:16:33.318428 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:16:33.319037 (+ 609us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:16:33.319342 (+ 305us) server_negotiation.cc:1035] Waiting for connection context
0928 02:16:33.320159 (+ 817us) server_negotiation.cc:299] Negotiation successful
0928 02:16:33.320493 (+ 334us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260928 02:16:33.326329 29243 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 02:16:33.333081 29243 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 02:16:33.343191 29243 catalog_manager.cc:1347] Generated new cluster ID: e95bce11ac294c74971d9250c1ee89cb
I20260928 02:16:33.343349 29243 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 02:16:33.355500 29243 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 02:16:33.357214 29243 catalog_manager.cc:1504] Loading token signing keys...
I20260928 02:16:33.378677 29243 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ab9c89d727134b53b0f4b2cb856e0364: Generated new TSK 0
I20260928 02:16:33.379698 29243 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 02:16:33.388885 7085 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskTlaUuc/build/asan/bin/kudu pid 29161
I20260928 02:16:33.390106 29223 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 02:16:33.484997 7085 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTlaUuc/build/asan/bin/kudu with pid 29161
2026-09-28T02:16:33Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/15 (2070 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/16
Loading random data
Initializing database '/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790561432251112-7085-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 02:16:33 dist-test-slave-86gv krb5kdc[29260](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 02:16:33 dist-test-slave-86gv krb5kdc[29260](info): set up 2 sockets
Sep 28 02:16:33 dist-test-slave-86gv krb5kdc[29260](info): commencing operation
krb5kdc: starting...
W20260928 02:16:34.621862 7085 mini_kdc.cc:121] Time spent starting KDC: real 1.100s user 0.010s sys 0.021s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790561432251112-7085-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790561432251112-7085-0/krb5kdc/test-user.keytab.
Sep 28 02:16:34 dist-test-slave-86gv krb5kdc[29260](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561794, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T02:16:34Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T02:16:34Z Disabled control of system clock
WARNING: no policy specified for kudu/127.6.235.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.235.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.235.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790561432251112-7085-0/krb5kdc/kudu_127.6.235.126.keytab.
Entry for principal kudu/127.6.235.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790561432251112-7085-0/krb5kdc/kudu_127.6.235.126.keytab.
I20260928 02:16:34.905336 7085 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTlaUuc/build/asan/bin/kudu
/tmp/dist-test-taskTlaUuc/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790561432251112-7085-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790561432251112-7085-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790561432251112-7085-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790561432251112-7085-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.235.126:34265
--webserver_interface=127.6.235.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790561432251112-7085-0/krb5kdc/kudu_127.6.235.126.keytab
--principal=kudu/127.6.235.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.235.84:37091
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.235.126:34265
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790561432251112-7085-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790561432251112-7085-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790561432251112-7085-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:16:35.262893 29273 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:16:35.263203 29273 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:16:35.263389 29273 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:16:35.270918 29273 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 02:16:35.271044 29273 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:16:35.271155 29273 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:16:35.271236 29273 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 02:16:35.271304 29273 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 02:16:35.281816 29273 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.235.84:37091
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790561432251112-7085-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790561432251112-7085-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.235.126:34265
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790561432251112-7085-0/krb5kdc/kudu_127.6.235.126.keytab
--principal=kudu/127.6.235.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.235.126:34265
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790561432251112-7085-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.235.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790561432251112-7085-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 00:51:59 UTC on ccycloud.quasar-axgkwn.root.comops.site
build id 4676464
ASAN enabled
I20260928 02:16:35.283226 29273 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:16:35.284628 29273 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:16:35.296761 29279 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:16:35.296844 29278 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:16:35.296866 29281 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:16:35.297701 29273 server_base.cc:1034] running on GCE node
I20260928 02:16:35.298544 29273 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:16:35.300151 29273 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:16:35.301417 29273 hybrid_clock.cc:648] HybridClock initialized: now 1790561795301374 us; error 36 us; skew 500 ppm
I20260928 02:16:35.301800 29273 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 02:16:35 dist-test-slave-86gv krb5kdc[29260](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561795, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.235.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:16:35.305217 29273 init.cc:377] Logged in from keytab as kudu/127.6.235.126@KRBTEST.COM (short username kudu)
I20260928 02:16:35.306839 29273 webserver.cc:460] Webserver started at http://127.6.235.126:46499/ using document root <none> and password file <none>
I20260928 02:16:35.307562 29273 fs_manager.cc:359] Metadata directory not provided
I20260928 02:16:35.307708 29273 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:16:35.308018 29273 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:16:35.311249 29273 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790561432251112-7085-0/minicluster-data/master-0/data/instance:
uuid: "97a72c3f53634b07a3927636bf05e877"
format_stamp: "Formatted at 2026-09-28 02:16:35 on dist-test-slave-86gv"
server_key: "b19aaab2ef381f2ab1a590f5094677db"
server_key_iv: "3a21ef9d39bf2ee13416e79b1cd5cd2c"
server_key_version: "encryptionkey@0"
I20260928 02:16:35.312041 29273 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790561432251112-7085-0/minicluster-data/master-0/wal/instance:
uuid: "97a72c3f53634b07a3927636bf05e877"
format_stamp: "Formatted at 2026-09-28 02:16:35 on dist-test-slave-86gv"
server_key: "b19aaab2ef381f2ab1a590f5094677db"
server_key_iv: "3a21ef9d39bf2ee13416e79b1cd5cd2c"
server_key_version: "encryptionkey@0"
I20260928 02:16:35.317520 29273 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260928 02:16:35.321476 29288 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:16:35.323292 29273 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260928 02:16:35.323523 29273 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790561432251112-7085-0/minicluster-data/master-0/data,/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790561432251112-7085-0/minicluster-data/master-0/wal
uuid: "97a72c3f53634b07a3927636bf05e877"
format_stamp: "Formatted at 2026-09-28 02:16:35 on dist-test-slave-86gv"
server_key: "b19aaab2ef381f2ab1a590f5094677db"
server_key_iv: "3a21ef9d39bf2ee13416e79b1cd5cd2c"
server_key_version: "encryptionkey@0"
I20260928 02:16:35.323766 29273 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790561432251112-7085-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790561432251112-7085-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790561432251112-7085-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:16:35.345736 29273 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:16:35.350589 29273 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:16:35.350936 29273 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:16:35.371119 29273 rpc_server.cc:307] RPC server started. Bound to: 127.6.235.126:34265
I20260928 02:16:35.371138 29339 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.235.126:34265 every 8 connection(s)
I20260928 02:16:35.372584 29273 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790561432251112-7085-0/minicluster-data/master-0/data/info.pb
I20260928 02:16:35.376622 29340 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:16:35.382848 7085 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTlaUuc/build/asan/bin/kudu as pid 29273
I20260928 02:16:35.383047 7085 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790561432251112-7085-0/minicluster-data/master-0/wal/instance
I20260928 02:16:35.383560 7085 external_mini_cluster.cc:1428] Setting key 9bb08098c51235009b8fbadf236c5df1
I20260928 02:16:35.388859 29340 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 97a72c3f53634b07a3927636bf05e877: Bootstrap starting.
I20260928 02:16:35.392251 29340 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 97a72c3f53634b07a3927636bf05e877: Neither blocks nor log segments found. Creating new log.
I20260928 02:16:35.393455 29340 log.cc:824] T 00000000000000000000000000000000 P 97a72c3f53634b07a3927636bf05e877: Log is configured to *not* fsync() on all Append() calls
I20260928 02:16:35.396648 29340 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 97a72c3f53634b07a3927636bf05e877: No bootstrap required, opened a new log
I20260928 02:16:35.397655 29343 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:16:35.385856 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39301 (local address 127.6.235.126:34265)
0928 02:16:35.386769 (+ 913us) server_negotiation.cc:206] Beginning negotiation
0928 02:16:35.386782 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0928 02:16:35.386834 (+ 52us) server_negotiation.cc:407] Connection header received
0928 02:16:35.388558 (+ 1724us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:16:35.388583 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:16:35.389150 (+ 567us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:16:35.389580 (+ 430us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:16:35.390128 (+ 548us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:16:35.391499 (+ 1371us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:16:35.392913 (+ 1414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:16:35.393648 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:16:35.393848 (+ 200us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:16:35.395171 (+ 1323us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:16:35.395179 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:16:35.395195 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 02:16:35.395789 (+ 594us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:16:35.395870 (+ 81us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 02:16:35.395895 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:16:35.396040 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0928 02:16:35.396267 (+ 227us) server_negotiation.cc:299] Negotiation successful
0928 02:16:35.396544 (+ 277us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":670,"thread_start_us":411,"threads_started":1}
I20260928 02:16:35.401562 7085 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 02:16:35.405397 29340 raft_consensus.cc:348] T 00000000000000000000000000000000 P 97a72c3f53634b07a3927636bf05e877 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "97a72c3f53634b07a3927636bf05e877" member_type: VOTER last_known_addr { host: "127.6.235.126" port: 34265 } }
I20260928 02:16:35.405995 29340 raft_consensus.cc:374] T 00000000000000000000000000000000 P 97a72c3f53634b07a3927636bf05e877 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:16:35.406221 29340 raft_consensus.cc:729] T 00000000000000000000000000000000 P 97a72c3f53634b07a3927636bf05e877 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 97a72c3f53634b07a3927636bf05e877, State: Initialized, Role: FOLLOWER
I20260928 02:16:35.406975 29340 consensus_queue.cc:260] T 00000000000000000000000000000000 P 97a72c3f53634b07a3927636bf05e877 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "97a72c3f53634b07a3927636bf05e877" member_type: VOTER last_known_addr { host: "127.6.235.126" port: 34265 } }
I20260928 02:16:35.407284 29340 raft_consensus.cc:388] T 00000000000000000000000000000000 P 97a72c3f53634b07a3927636bf05e877 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 02:16:35.407430 29340 raft_consensus.cc:482] T 00000000000000000000000000000000 P 97a72c3f53634b07a3927636bf05e877 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 02:16:35.407608 29340 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 97a72c3f53634b07a3927636bf05e877 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:16:35.410099 29340 raft_consensus.cc:504] T 00000000000000000000000000000000 P 97a72c3f53634b07a3927636bf05e877 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "97a72c3f53634b07a3927636bf05e877" member_type: VOTER last_known_addr { host: "127.6.235.126" port: 34265 } }
I20260928 02:16:35.410705 29340 leader_election.cc:302] T 00000000000000000000000000000000 P 97a72c3f53634b07a3927636bf05e877 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 97a72c3f53634b07a3927636bf05e877; no voters:
I20260928 02:16:35.411556 29340 leader_election.cc:288] T 00000000000000000000000000000000 P 97a72c3f53634b07a3927636bf05e877 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 02:16:35.411666 29350 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 97a72c3f53634b07a3927636bf05e877 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:16:35.412947 29343 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:16:35.406043 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55741 (local address 127.6.235.126:34265)
0928 02:16:35.406261 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0928 02:16:35.406268 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 02:16:35.406665 (+ 397us) server_negotiation.cc:407] Connection header received
0928 02:16:35.406907 (+ 242us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:16:35.406911 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:16:35.407001 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:16:35.407105 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:16:35.407689 (+ 584us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:16:35.408485 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:16:35.409839 (+ 1354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:16:35.410411 (+ 572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:16:35.410540 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:16:35.411732 (+ 1192us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:16:35.411738 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:16:35.411744 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 02:16:35.411770 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:16:35.411858 (+ 88us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 02:16:35.411881 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:16:35.412001 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0928 02:16:35.412292 (+ 291us) server_negotiation.cc:299] Negotiation successful
0928 02:16:35.412440 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260928 02:16:35.412950 29350 raft_consensus.cc:686] T 00000000000000000000000000000000 P 97a72c3f53634b07a3927636bf05e877 [term 1 LEADER]: Becoming Leader. State: Replica: 97a72c3f53634b07a3927636bf05e877, State: Running, Role: LEADER
I20260928 02:16:35.413888 29350 consensus_queue.cc:237] T 00000000000000000000000000000000 P 97a72c3f53634b07a3927636bf05e877 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "97a72c3f53634b07a3927636bf05e877" member_type: VOTER last_known_addr { host: "127.6.235.126" port: 34265 } }
I20260928 02:16:35.414212 29340 sys_catalog.cc:564] T 00000000000000000000000000000000 P 97a72c3f53634b07a3927636bf05e877 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 02:16:35.417099 29352 sys_catalog.cc:455] T 00000000000000000000000000000000 P 97a72c3f53634b07a3927636bf05e877 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 97a72c3f53634b07a3927636bf05e877. Latest consensus state: current_term: 1 leader_uuid: "97a72c3f53634b07a3927636bf05e877" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "97a72c3f53634b07a3927636bf05e877" member_type: VOTER last_known_addr { host: "127.6.235.126" port: 34265 } } }
I20260928 02:16:35.417604 29352 sys_catalog.cc:458] T 00000000000000000000000000000000 P 97a72c3f53634b07a3927636bf05e877 [sys.catalog]: This master's current role is: LEADER
I20260928 02:16:35.417121 29351 sys_catalog.cc:455] T 00000000000000000000000000000000 P 97a72c3f53634b07a3927636bf05e877 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "97a72c3f53634b07a3927636bf05e877" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "97a72c3f53634b07a3927636bf05e877" member_type: VOTER last_known_addr { host: "127.6.235.126" port: 34265 } } }
I20260928 02:16:35.418201 29351 sys_catalog.cc:458] T 00000000000000000000000000000000 P 97a72c3f53634b07a3927636bf05e877 [sys.catalog]: This master's current role is: LEADER
I20260928 02:16:35.419054 29356 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 02:16:35.424723 29356 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 02:16:35.435128 29356 catalog_manager.cc:1347] Generated new cluster ID: 9e2e1a81c0ae4d30ad4701e04d3e6bb8
I20260928 02:16:35.435268 29356 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 02:16:35.443610 29356 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 02:16:35.444983 29356 catalog_manager.cc:1504] Loading token signing keys...
I20260928 02:16:35.463934 29356 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 97a72c3f53634b07a3927636bf05e877: Generated new TSK 0
I20260928 02:16:35.464984 29356 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 02:16:35.474200 7085 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskTlaUuc/build/asan/bin/kudu pid 29273
W20260928 02:16:35.475534 29335 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:39301
W20260928 02:16:35.476284 7085 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskTlaUuc/build/asan/bin/kudu pid 29273: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260928 02:16:35.476445 7085 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTlaUuc/build/asan/bin/kudu with pid 29273
2026-09-28T02:16:35Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/16 (1991 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/17
Loading random data
Initializing database '/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790561432251112-7085-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 02:16:35 dist-test-slave-86gv krb5kdc[29371](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 02:16:35 dist-test-slave-86gv krb5kdc[29371](info): set up 2 sockets
Sep 28 02:16:35 dist-test-slave-86gv krb5kdc[29371](info): commencing operation
krb5kdc: starting...
W20260928 02:16:36.601854 7085 mini_kdc.cc:121] Time spent starting KDC: real 1.088s user 0.003s sys 0.027s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790561432251112-7085-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790561432251112-7085-0/krb5kdc/test-user.keytab.
Sep 28 02:16:36 dist-test-slave-86gv krb5kdc[29371](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561796, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T02:16:36Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T02:16:36Z Disabled control of system clock
WARNING: no policy specified for kudu/127.6.235.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.235.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.235.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790561432251112-7085-0/krb5kdc/kudu_127.6.235.126.keytab.
Entry for principal kudu/127.6.235.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790561432251112-7085-0/krb5kdc/kudu_127.6.235.126.keytab.
I20260928 02:16:36.893301 7085 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTlaUuc/build/asan/bin/kudu
/tmp/dist-test-taskTlaUuc/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790561432251112-7085-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790561432251112-7085-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790561432251112-7085-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790561432251112-7085-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.235.126:34571
--webserver_interface=127.6.235.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790561432251112-7085-0/krb5kdc/kudu_127.6.235.126.keytab
--principal=kudu/127.6.235.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.235.84:37909
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.235.126:34571
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790561432251112-7085-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790561432251112-7085-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790561432251112-7085-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:16:37.247504 29384 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:16:37.247833 29384 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:16:37.248006 29384 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:16:37.255631 29384 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 02:16:37.255770 29384 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:16:37.255852 29384 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:16:37.255962 29384 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 02:16:37.256036 29384 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 02:16:37.266417 29384 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.235.84:37909
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790561432251112-7085-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790561432251112-7085-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.235.126:34571
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790561432251112-7085-0/krb5kdc/kudu_127.6.235.126.keytab
--principal=kudu/127.6.235.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.235.126:34571
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790561432251112-7085-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.235.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790561432251112-7085-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 00:51:59 UTC on ccycloud.quasar-axgkwn.root.comops.site
build id 4676464
ASAN enabled
I20260928 02:16:37.267828 29384 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:16:37.269244 29384 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:16:37.281522 29392 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:16:37.281570 29390 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:16:37.281543 29389 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:16:37.282380 29384 server_base.cc:1034] running on GCE node
I20260928 02:16:37.283315 29384 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:16:37.284860 29384 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:16:37.286144 29384 hybrid_clock.cc:648] HybridClock initialized: now 1790561797286088 us; error 51 us; skew 500 ppm
I20260928 02:16:37.286593 29384 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 02:16:37 dist-test-slave-86gv krb5kdc[29371](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561797, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.235.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:16:37.296496 29384 init.cc:377] Logged in from keytab as kudu/127.6.235.126@KRBTEST.COM (short username kudu)
I20260928 02:16:37.298354 29384 webserver.cc:460] Webserver started at http://127.6.235.126:41939/ using document root <none> and password file <none>
I20260928 02:16:37.299149 29384 fs_manager.cc:359] Metadata directory not provided
I20260928 02:16:37.299299 29384 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:16:37.299628 29384 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:16:37.302973 29384 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790561432251112-7085-0/minicluster-data/master-0/data/instance:
uuid: "76b94978ce6a46cbaf7efa05588aec2a"
format_stamp: "Formatted at 2026-09-28 02:16:37 on dist-test-slave-86gv"
server_key: "18ee8ea7171252a1767443e097eefe02"
server_key_iv: "efcef04f104828d37626f572ad05779f"
server_key_version: "encryptionkey@0"
I20260928 02:16:37.303879 29384 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790561432251112-7085-0/minicluster-data/master-0/wal/instance:
uuid: "76b94978ce6a46cbaf7efa05588aec2a"
format_stamp: "Formatted at 2026-09-28 02:16:37 on dist-test-slave-86gv"
server_key: "18ee8ea7171252a1767443e097eefe02"
server_key_iv: "efcef04f104828d37626f572ad05779f"
server_key_version: "encryptionkey@0"
I20260928 02:16:37.309998 29384 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.000s
I20260928 02:16:37.314087 29399 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:16:37.315682 29384 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260928 02:16:37.315900 29384 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790561432251112-7085-0/minicluster-data/master-0/data,/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790561432251112-7085-0/minicluster-data/master-0/wal
uuid: "76b94978ce6a46cbaf7efa05588aec2a"
format_stamp: "Formatted at 2026-09-28 02:16:37 on dist-test-slave-86gv"
server_key: "18ee8ea7171252a1767443e097eefe02"
server_key_iv: "efcef04f104828d37626f572ad05779f"
server_key_version: "encryptionkey@0"
I20260928 02:16:37.316133 29384 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790561432251112-7085-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790561432251112-7085-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790561432251112-7085-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:16:37.322497 29384 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:16:37.327060 29384 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:16:37.327386 29384 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:16:37.347327 29384 rpc_server.cc:307] RPC server started. Bound to: 127.6.235.126:34571
I20260928 02:16:37.347350 29450 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.235.126:34571 every 8 connection(s)
I20260928 02:16:37.348846 29384 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790561432251112-7085-0/minicluster-data/master-0/data/info.pb
I20260928 02:16:37.351637 7085 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTlaUuc/build/asan/bin/kudu as pid 29384
I20260928 02:16:37.351825 7085 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790561432251112-7085-0/minicluster-data/master-0/wal/instance
I20260928 02:16:37.352375 7085 external_mini_cluster.cc:1428] Setting key 32c4a48d3d38788b5c5e69cabdc4d428
I20260928 02:16:37.353773 29451 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:16:37.360512 29454 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:16:37.354291 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44293 (local address 127.6.235.126:34571)
0928 02:16:37.355089 (+ 798us) server_negotiation.cc:206] Beginning negotiation
0928 02:16:37.355101 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0928 02:16:37.355145 (+ 44us) server_negotiation.cc:407] Connection header received
0928 02:16:37.356721 (+ 1576us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:16:37.356745 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:16:37.357214 (+ 469us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:16:37.357584 (+ 370us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:16:37.358689 (+ 1105us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:16:37.358696 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:16:37.358713 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 02:16:37.358729 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:16:37.358856 (+ 127us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 02:16:37.358878 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:16:37.359065 (+ 187us) server_negotiation.cc:1035] Waiting for connection context
0928 02:16:37.359178 (+ 113us) server_negotiation.cc:299] Negotiation successful
0928 02:16:37.359495 (+ 317us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":576,"thread_start_us":359,"threads_started":1}
I20260928 02:16:37.364573 7085 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 02:16:37.368022 29451 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 76b94978ce6a46cbaf7efa05588aec2a: Bootstrap starting.
I20260928 02:16:37.372551 29451 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 76b94978ce6a46cbaf7efa05588aec2a: Neither blocks nor log segments found. Creating new log.
I20260928 02:16:37.372848 29454 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:16:37.370035 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41717 (local address 127.6.235.126:34571)
0928 02:16:37.370269 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0928 02:16:37.370275 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 02:16:37.370572 (+ 297us) server_negotiation.cc:407] Connection header received
0928 02:16:37.370974 (+ 402us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:16:37.370979 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:16:37.371035 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:16:37.371126 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:16:37.372109 (+ 983us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:16:37.372114 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:16:37.372117 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 02:16:37.372136 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:16:37.372182 (+ 46us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 02:16:37.372197 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:16:37.372290 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0928 02:16:37.372445 (+ 155us) server_negotiation.cc:299] Negotiation successful
0928 02:16:37.372493 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":96}
I20260928 02:16:37.373895 29451 log.cc:824] T 00000000000000000000000000000000 P 76b94978ce6a46cbaf7efa05588aec2a: Log is configured to *not* fsync() on all Append() calls
I20260928 02:16:37.377264 29451 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 76b94978ce6a46cbaf7efa05588aec2a: No bootstrap required, opened a new log
I20260928 02:16:37.385406 29451 raft_consensus.cc:348] T 00000000000000000000000000000000 P 76b94978ce6a46cbaf7efa05588aec2a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "76b94978ce6a46cbaf7efa05588aec2a" member_type: VOTER last_known_addr { host: "127.6.235.126" port: 34571 } }
I20260928 02:16:37.385859 29451 raft_consensus.cc:374] T 00000000000000000000000000000000 P 76b94978ce6a46cbaf7efa05588aec2a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:16:37.385983 29451 raft_consensus.cc:729] T 00000000000000000000000000000000 P 76b94978ce6a46cbaf7efa05588aec2a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 76b94978ce6a46cbaf7efa05588aec2a, State: Initialized, Role: FOLLOWER
I20260928 02:16:37.386710 29451 consensus_queue.cc:260] T 00000000000000000000000000000000 P 76b94978ce6a46cbaf7efa05588aec2a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "76b94978ce6a46cbaf7efa05588aec2a" member_type: VOTER last_known_addr { host: "127.6.235.126" port: 34571 } }
I20260928 02:16:37.387008 29451 raft_consensus.cc:388] T 00000000000000000000000000000000 P 76b94978ce6a46cbaf7efa05588aec2a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 02:16:37.387183 29451 raft_consensus.cc:482] T 00000000000000000000000000000000 P 76b94978ce6a46cbaf7efa05588aec2a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 02:16:37.387387 29451 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 76b94978ce6a46cbaf7efa05588aec2a [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:16:37.389798 29451 raft_consensus.cc:504] T 00000000000000000000000000000000 P 76b94978ce6a46cbaf7efa05588aec2a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "76b94978ce6a46cbaf7efa05588aec2a" member_type: VOTER last_known_addr { host: "127.6.235.126" port: 34571 } }
I20260928 02:16:37.390355 29451 leader_election.cc:302] T 00000000000000000000000000000000 P 76b94978ce6a46cbaf7efa05588aec2a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 76b94978ce6a46cbaf7efa05588aec2a; no voters:
I20260928 02:16:37.391044 29451 leader_election.cc:288] T 00000000000000000000000000000000 P 76b94978ce6a46cbaf7efa05588aec2a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 02:16:37.391187 29461 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 76b94978ce6a46cbaf7efa05588aec2a [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:16:37.391875 29461 raft_consensus.cc:686] T 00000000000000000000000000000000 P 76b94978ce6a46cbaf7efa05588aec2a [term 1 LEADER]: Becoming Leader. State: Replica: 76b94978ce6a46cbaf7efa05588aec2a, State: Running, Role: LEADER
I20260928 02:16:37.392563 29461 consensus_queue.cc:237] T 00000000000000000000000000000000 P 76b94978ce6a46cbaf7efa05588aec2a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "76b94978ce6a46cbaf7efa05588aec2a" member_type: VOTER last_known_addr { host: "127.6.235.126" port: 34571 } }
I20260928 02:16:37.393435 29451 sys_catalog.cc:564] T 00000000000000000000000000000000 P 76b94978ce6a46cbaf7efa05588aec2a [sys.catalog]: configured and running, proceeding with master startup.
I20260928 02:16:37.395443 29462 sys_catalog.cc:455] T 00000000000000000000000000000000 P 76b94978ce6a46cbaf7efa05588aec2a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "76b94978ce6a46cbaf7efa05588aec2a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "76b94978ce6a46cbaf7efa05588aec2a" member_type: VOTER last_known_addr { host: "127.6.235.126" port: 34571 } } }
I20260928 02:16:37.395921 29462 sys_catalog.cc:458] T 00000000000000000000000000000000 P 76b94978ce6a46cbaf7efa05588aec2a [sys.catalog]: This master's current role is: LEADER
I20260928 02:16:37.395838 29463 sys_catalog.cc:455] T 00000000000000000000000000000000 P 76b94978ce6a46cbaf7efa05588aec2a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 76b94978ce6a46cbaf7efa05588aec2a. Latest consensus state: current_term: 1 leader_uuid: "76b94978ce6a46cbaf7efa05588aec2a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "76b94978ce6a46cbaf7efa05588aec2a" member_type: VOTER last_known_addr { host: "127.6.235.126" port: 34571 } } }
I20260928 02:16:37.396111 29463 sys_catalog.cc:458] T 00000000000000000000000000000000 P 76b94978ce6a46cbaf7efa05588aec2a [sys.catalog]: This master's current role is: LEADER
I20260928 02:16:37.398053 29469 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 02:16:37.402906 29469 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 02:16:37.411855 29469 catalog_manager.cc:1347] Generated new cluster ID: 8cd4d102331243218b6139bc136a486a
I20260928 02:16:37.412003 29469 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 02:16:37.425302 29469 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 02:16:37.425774 29469 catalog_manager.cc:1504] Loading token signing keys...
I20260928 02:16:37.433678 29469 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 76b94978ce6a46cbaf7efa05588aec2a: Generated new TSK 0
I20260928 02:16:37.434613 29469 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 02:16:37.450878 7085 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskTlaUuc/build/asan/bin/kudu pid 29384
W20260928 02:16:37.451980 29446 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:44293
W20260928 02:16:37.452701 7085 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskTlaUuc/build/asan/bin/kudu pid 29384: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260928 02:16:37.452845 7085 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTlaUuc/build/asan/bin/kudu with pid 29384
2026-09-28T02:16:37Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/17 (1977 ms)
[----------] 18 tests from AuthTokenIssuingTest (36619 ms total)
[----------] 2 tests from ConnectToFollowerMasterTest
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0
Loading random data
Initializing database '/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790561432251112-7085-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 02:16:37 dist-test-slave-86gv krb5kdc[29482](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 02:16:37 dist-test-slave-86gv krb5kdc[29482](info): set up 2 sockets
Sep 28 02:16:37 dist-test-slave-86gv krb5kdc[29482](info): commencing operation
krb5kdc: starting...
W20260928 02:16:38.598559 7085 mini_kdc.cc:121] Time spent starting KDC: real 1.107s user 0.000s sys 0.030s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790561432251112-7085-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790561432251112-7085-0/krb5kdc/test-user.keytab.
Sep 28 02:16:38 dist-test-slave-86gv krb5kdc[29482](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561798, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T02:16:38Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T02:16:38Z Disabled control of system clock
WARNING: no policy specified for kudu/127.6.235.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.235.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.235.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790561432251112-7085-0/krb5kdc/kudu_127.6.235.126.keytab.
Entry for principal kudu/127.6.235.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790561432251112-7085-0/krb5kdc/kudu_127.6.235.126.keytab.
I20260928 02:16:38.880136 7085 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTlaUuc/build/asan/bin/kudu
/tmp/dist-test-taskTlaUuc/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790561432251112-7085-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790561432251112-7085-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790561432251112-7085-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790561432251112-7085-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.235.126:33829
--webserver_interface=127.6.235.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790561432251112-7085-0/krb5kdc/kudu_127.6.235.126.keytab
--principal=kudu/127.6.235.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.235.84:34081
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.235.126:33829,127.6.235.125:32979,127.6.235.124:34699
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790561432251112-7085-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790561432251112-7085-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790561432251112-7085-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:16:39.231528 29495 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260928 02:16:39.231920 29495 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:16:39.231992 29495 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:16:39.232084 29495 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:16:39.239830 29495 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 02:16:39.240015 29495 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:16:39.240156 29495 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:16:39.240255 29495 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 02:16:39.240382 29495 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 02:16:39.251020 29495 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.235.84:34081
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790561432251112-7085-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790561432251112-7085-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.235.126:33829,127.6.235.125:32979,127.6.235.124:34699
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790561432251112-7085-0/krb5kdc/kudu_127.6.235.126.keytab
--principal=kudu/127.6.235.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.235.126:33829
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790561432251112-7085-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.235.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790561432251112-7085-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 00:51:59 UTC on ccycloud.quasar-axgkwn.root.comops.site
build id 4676464
ASAN enabled
I20260928 02:16:39.252562 29495 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:16:39.254016 29495 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:16:39.266700 29501 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:16:39.266778 29503 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:16:39.266758 29500 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:16:39.267590 29495 server_base.cc:1034] running on GCE node
I20260928 02:16:39.268551 29495 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:16:39.270329 29495 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:16:39.271612 29495 hybrid_clock.cc:648] HybridClock initialized: now 1790561799271549 us; error 40 us; skew 500 ppm
I20260928 02:16:39.271997 29495 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 02:16:39 dist-test-slave-86gv krb5kdc[29482](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561799, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.235.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:16:39.275324 29495 init.cc:377] Logged in from keytab as kudu/127.6.235.126@KRBTEST.COM (short username kudu)
I20260928 02:16:39.277050 29495 webserver.cc:460] Webserver started at http://127.6.235.126:42889/ using document root <none> and password file <none>
I20260928 02:16:39.277765 29495 fs_manager.cc:359] Metadata directory not provided
I20260928 02:16:39.277892 29495 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:16:39.278213 29495 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:16:39.281314 29495 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790561432251112-7085-0/minicluster-data/master-0/data/instance:
uuid: "f48dda22055547a596b2e2575946f3a4"
format_stamp: "Formatted at 2026-09-28 02:16:39 on dist-test-slave-86gv"
server_key: "5c60b927e6d8275a988f0af0f8363783"
server_key_iv: "c3c2ed62d58c70e0ae73497080e6ff83"
server_key_version: "encryptionkey@0"
I20260928 02:16:39.282138 29495 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790561432251112-7085-0/minicluster-data/master-0/wal/instance:
uuid: "f48dda22055547a596b2e2575946f3a4"
format_stamp: "Formatted at 2026-09-28 02:16:39 on dist-test-slave-86gv"
server_key: "5c60b927e6d8275a988f0af0f8363783"
server_key_iv: "c3c2ed62d58c70e0ae73497080e6ff83"
server_key_version: "encryptionkey@0"
I20260928 02:16:39.287616 29495 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.005s
I20260928 02:16:39.291290 29510 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:16:39.292872 29495 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260928 02:16:39.293138 29495 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790561432251112-7085-0/minicluster-data/master-0/data,/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790561432251112-7085-0/minicluster-data/master-0/wal
uuid: "f48dda22055547a596b2e2575946f3a4"
format_stamp: "Formatted at 2026-09-28 02:16:39 on dist-test-slave-86gv"
server_key: "5c60b927e6d8275a988f0af0f8363783"
server_key_iv: "c3c2ed62d58c70e0ae73497080e6ff83"
server_key_version: "encryptionkey@0"
I20260928 02:16:39.293363 29495 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790561432251112-7085-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790561432251112-7085-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790561432251112-7085-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:16:39.312361 29495 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:16:39.317103 29495 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:16:39.317392 29495 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:16:39.336766 29495 rpc_server.cc:307] RPC server started. Bound to: 127.6.235.126:33829
I20260928 02:16:39.336932 29561 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.235.126:33829 every 8 connection(s)
I20260928 02:16:39.338320 29495 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790561432251112-7085-0/minicluster-data/master-0/data/info.pb
I20260928 02:16:39.342549 29562 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:16:39.347347 7085 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTlaUuc/build/asan/bin/kudu as pid 29495
I20260928 02:16:39.347570 7085 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790561432251112-7085-0/minicluster-data/master-0/wal/instance
I20260928 02:16:39.348238 7085 external_mini_cluster.cc:1428] Setting key 764a930dccf20d70b2a520dad21c1da9
I20260928 02:16:39.349535 29562 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.6.235.126" port: 33829 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 02:16:39 dist-test-slave-86gv krb5kdc[29482](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561799, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.235.126@KRBTEST.COM for kudu/127.6.235.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.6.235.125@KRBTEST.COM; defaulting to no policy
I20260928 02:16:39.377167 29564 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:16:39.353125 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.235.126:33829 (local address 127.0.0.1:41312)
0928 02:16:39.354155 (+ 1030us) negotiation.cc:107] Waiting for socket to connect
0928 02:16:39.354208 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0928 02:16:39.356101 (+ 1893us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:16:39.357311 (+ 1210us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:16:39.357324 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:16:39.358231 (+ 907us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:16:39.358775 (+ 544us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:16:39.358807 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:16:39.360617 (+ 1810us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:16:39.360625 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:16:39.362650 (+ 2025us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:16:39.362673 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:16:39.363727 (+ 1054us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:16:39.363737 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:16:39.363976 (+ 239us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:16:39.364776 (+ 800us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:16:39.364813 (+ 37us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:16:39.368025 (+ 3212us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:16:39.372318 (+ 4293us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:16:39.372336 (+ 18us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:16:39.372354 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:16:39.373110 (+ 756us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:16:39.373612 (+ 502us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:16:39.373619 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:16:39.373623 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:16:39.373852 (+ 229us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:16:39.374679 (+ 827us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:16:39.374690 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:16:39.374995 (+ 305us) client_negotiation.cc:769] Sending connection context
0928 02:16:39.375345 (+ 350us) client_negotiation.cc:240] Negotiation successful
0928 02:16:39.376028 (+ 683us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":715,"mutex_wait_us":113,"thread_start_us":454,"threads_started":1}
I20260928 02:16:39.377198 29565 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:16:39.353125 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41312 (local address 127.6.235.126:33829)
0928 02:16:39.354220 (+ 1095us) server_negotiation.cc:206] Beginning negotiation
0928 02:16:39.354231 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 02:16:39.354481 (+ 250us) server_negotiation.cc:407] Connection header received
0928 02:16:39.356567 (+ 2086us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:16:39.356579 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:16:39.357000 (+ 421us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:16:39.357189 (+ 189us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:16:39.359038 (+ 1849us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:16:39.360394 (+ 1356us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:16:39.362847 (+ 2453us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:16:39.363457 (+ 610us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:16:39.363608 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:16:39.368324 (+ 4716us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:16:39.368333 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:16:39.368348 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:16:39.368391 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:16:39.372124 (+ 3733us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:16:39.373301 (+ 1177us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:16:39.373308 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:16:39.373310 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:16:39.373481 (+ 171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:16:39.373970 (+ 489us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:16:39.373973 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:16:39.373975 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:16:39.374525 (+ 550us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:16:39.374698 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0928 02:16:39.375464 (+ 766us) server_negotiation.cc:299] Negotiation successful
0928 02:16:39.375799 (+ 335us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":832,"thread_start_us":1025,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.235.125@KRBTEST.COM" created.
I20260928 02:16:39.380738 29562 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.6.235.125" port: 32979 } has no permanent_uuid. Determining permanent_uuid...
I20260928 02:16:39.382133 29564 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:16:39.381595 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.235.125:32979 (local address 127.0.0.1:58990)
0928 02:16:39.381828 (+ 233us) negotiation.cc:107] Waiting for socket to connect
0928 02:16:39.381966 (+ 138us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.235.125:32979: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
W20260928 02:16:39.382483 29511 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.235.125:32979: connect: Connection refused (error 111)
I20260928 02:16:39.384582 29564 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:16:39.384097 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.235.125:32979 (local address 127.0.0.1:59000)
0928 02:16:39.384292 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0928 02:16:39.384391 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.235.125:32979: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
W20260928 02:16:39.384869 29562 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.6.235.125:32979: Network error: Client connection negotiation failed: client connection to 127.6.235.125:32979: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.235.125 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790561432251112-7085-0/krb5kdc/kudu_127.6.235.125.keytab.
Entry for principal kudu/127.6.235.125 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790561432251112-7085-0/krb5kdc/kudu_127.6.235.125.keytab.
I20260928 02:16:39.412477 7085 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTlaUuc/build/asan/bin/kudu
/tmp/dist-test-taskTlaUuc/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790561432251112-7085-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790561432251112-7085-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790561432251112-7085-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790561432251112-7085-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.235.125:32979
--webserver_interface=127.6.235.125
--webserver_port=0
--keytab_file=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790561432251112-7085-0/krb5kdc/kudu_127.6.235.125.keytab
--principal=kudu/127.6.235.125
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.235.84:34081
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.235.126:33829,127.6.235.125:32979,127.6.235.124:34699
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790561432251112-7085-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790561432251112-7085-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790561432251112-7085-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 02:16:39.434329 29562 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.235.125" port: 32979 } attempt: 1
I20260928 02:16:39.435704 29564 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:16:39.435142 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.235.125:32979 (local address 127.0.0.1:59006)
0928 02:16:39.435381 (+ 239us) negotiation.cc:107] Waiting for socket to connect
0928 02:16:39.435532 (+ 151us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.235.125:32979: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":98}
I20260928 02:16:39.436977 29564 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:16:39.436457 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.235.125:32979 (local address 127.0.0.1:59014)
0928 02:16:39.436714 (+ 257us) negotiation.cc:107] Waiting for socket to connect
0928 02:16:39.436797 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.235.125:32979: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":149}
W20260928 02:16:39.437201 29562 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.6.235.125:32979: Network error: Client connection negotiation failed: client connection to 127.6.235.125:32979: connect: Connection refused (error 111)
I20260928 02:16:39.505501 29562 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.235.125" port: 32979 } attempt: 2
I20260928 02:16:39.506800 29564 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:16:39.506268 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.235.125:32979 (local address 127.0.0.1:59024)
0928 02:16:39.506552 (+ 284us) negotiation.cc:107] Waiting for socket to connect
0928 02:16:39.506645 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.235.125:32979: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":82}
I20260928 02:16:39.507941 29564 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:16:39.507534 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.235.125:32979 (local address 127.0.0.1:59040)
0928 02:16:39.507723 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0928 02:16:39.507801 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.235.125:32979: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
W20260928 02:16:39.508131 29562 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.6.235.125:32979: Network error: Client connection negotiation failed: client connection to 127.6.235.125:32979: connect: Connection refused (error 111)
I20260928 02:16:39.599381 29562 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.235.125" port: 32979 } attempt: 3
I20260928 02:16:39.600651 29564 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:16:39.600192 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.235.125:32979 (local address 127.0.0.1:59048)
0928 02:16:39.600402 (+ 210us) negotiation.cc:107] Waiting for socket to connect
0928 02:16:39.600500 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.235.125:32979: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":73}
I20260928 02:16:39.601820 29564 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:16:39.601405 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.235.125:32979 (local address 127.0.0.1:59064)
0928 02:16:39.601573 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0928 02:16:39.601650 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.235.125:32979: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
W20260928 02:16:39.602008 29562 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.6.235.125:32979: Network error: Client connection negotiation failed: client connection to 127.6.235.125:32979: connect: Connection refused (error 111)
I20260928 02:16:39.745278 29562 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.235.125" port: 32979 } attempt: 4
I20260928 02:16:39.746434 29564 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:16:39.745899 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.235.125:32979 (local address 127.0.0.1:59066)
0928 02:16:39.746204 (+ 305us) negotiation.cc:107] Waiting for socket to connect
0928 02:16:39.746296 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.235.125:32979: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":141}
I20260928 02:16:39.747486 29564 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:16:39.747117 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.235.125:32979 (local address 127.0.0.1:59082)
0928 02:16:39.747276 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0928 02:16:39.747362 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.235.125:32979: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260928 02:16:39.747663 29562 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.6.235.125:32979: Network error: Client connection negotiation failed: client connection to 127.6.235.125:32979: connect: Connection refused (error 111)
W20260928 02:16:39.762497 29569 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260928 02:16:39.762907 29569 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:16:39.763026 29569 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:16:39.763156 29569 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:16:39.770777 29569 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 02:16:39.770912 29569 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:16:39.771040 29569 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:16:39.771126 29569 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 02:16:39.771232 29569 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 02:16:39.781473 29569 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.235.84:34081
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790561432251112-7085-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790561432251112-7085-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.6.235.126:33829,127.6.235.125:32979,127.6.235.124:34699
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790561432251112-7085-0/krb5kdc/kudu_127.6.235.125.keytab
--principal=kudu/127.6.235.125
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.235.125:32979
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790561432251112-7085-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.6.235.125
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790561432251112-7085-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 00:51:59 UTC on ccycloud.quasar-axgkwn.root.comops.site
build id 4676464
ASAN enabled
I20260928 02:16:39.783003 29569 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:16:39.784338 29569 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:16:39.796612 29575 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:16:39.796711 29574 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:16:39.796737 29577 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:16:39.797355 29569 server_base.cc:1034] running on GCE node
I20260928 02:16:39.798336 29569 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:16:39.799849 29569 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:16:39.801167 29569 hybrid_clock.cc:648] HybridClock initialized: now 1790561799801090 us; error 45 us; skew 500 ppm
I20260928 02:16:39.801568 29569 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 02:16:39 dist-test-slave-86gv krb5kdc[29482](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561799, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.235.125@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:16:39.805510 29569 init.cc:377] Logged in from keytab as kudu/127.6.235.125@KRBTEST.COM (short username kudu)
I20260928 02:16:39.807180 29569 webserver.cc:460] Webserver started at http://127.6.235.125:33447/ using document root <none> and password file <none>
I20260928 02:16:39.807955 29569 fs_manager.cc:359] Metadata directory not provided
I20260928 02:16:39.808055 29569 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:16:39.808358 29569 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:16:39.811830 29569 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790561432251112-7085-0/minicluster-data/master-1/data/instance:
uuid: "aab697abb3f64c20a42d6b6c684ea39f"
format_stamp: "Formatted at 2026-09-28 02:16:39 on dist-test-slave-86gv"
server_key: "fe1cf749f116b2807b614e037d6baf92"
server_key_iv: "333fcb3054b8544f491008efb8815244"
server_key_version: "encryptionkey@0"
I20260928 02:16:39.812677 29569 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790561432251112-7085-0/minicluster-data/master-1/wal/instance:
uuid: "aab697abb3f64c20a42d6b6c684ea39f"
format_stamp: "Formatted at 2026-09-28 02:16:39 on dist-test-slave-86gv"
server_key: "fe1cf749f116b2807b614e037d6baf92"
server_key_iv: "333fcb3054b8544f491008efb8815244"
server_key_version: "encryptionkey@0"
I20260928 02:16:39.818442 29569 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260928 02:16:39.822355 29584 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:16:39.823843 29569 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260928 02:16:39.823994 29569 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790561432251112-7085-0/minicluster-data/master-1/data,/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790561432251112-7085-0/minicluster-data/master-1/wal
uuid: "aab697abb3f64c20a42d6b6c684ea39f"
format_stamp: "Formatted at 2026-09-28 02:16:39 on dist-test-slave-86gv"
server_key: "fe1cf749f116b2807b614e037d6baf92"
server_key_iv: "333fcb3054b8544f491008efb8815244"
server_key_version: "encryptionkey@0"
I20260928 02:16:39.824184 29569 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790561432251112-7085-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790561432251112-7085-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790561432251112-7085-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:16:39.841872 29569 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:16:39.845986 29569 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:16:39.846207 29569 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:16:39.864782 29569 rpc_server.cc:307] RPC server started. Bound to: 127.6.235.125:32979
I20260928 02:16:39.864804 29635 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.235.125:32979 every 8 connection(s)
I20260928 02:16:39.866217 29569 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790561432251112-7085-0/minicluster-data/master-1/data/info.pb
I20260928 02:16:39.869339 7085 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTlaUuc/build/asan/bin/kudu as pid 29569
I20260928 02:16:39.869529 7085 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790561432251112-7085-0/minicluster-data/master-1/wal/instance
I20260928 02:16:39.870060 7085 external_mini_cluster.cc:1428] Setting key d436dd63db3c98aa514b6429574185b8
I20260928 02:16:39.870594 29636 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:16:39.880831 29636 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.6.235.126" port: 33829 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 02:16:39 dist-test-slave-86gv krb5kdc[29482](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561799, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.235.125@KRBTEST.COM for kudu/127.6.235.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.6.235.124@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.235.124@KRBTEST.COM" created.
I20260928 02:16:39.903892 29639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:16:39.883805 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41324 (local address 127.6.235.126:33829)
0928 02:16:39.884334 (+ 529us) server_negotiation.cc:206] Beginning negotiation
0928 02:16:39.884343 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 02:16:39.885439 (+ 1096us) server_negotiation.cc:407] Connection header received
0928 02:16:39.887349 (+ 1910us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:16:39.887358 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:16:39.887455 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:16:39.887595 (+ 140us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:16:39.889197 (+ 1602us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:16:39.890369 (+ 1172us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:16:39.892019 (+ 1650us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:16:39.892833 (+ 814us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:16:39.893037 (+ 204us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:16:39.897725 (+ 4688us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:16:39.897730 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:16:39.897734 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:16:39.897784 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:16:39.900585 (+ 2801us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:16:39.901520 (+ 935us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:16:39.901524 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:16:39.901527 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:16:39.901651 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:16:39.902142 (+ 491us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:16:39.902145 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:16:39.902147 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:16:39.902417 (+ 270us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:16:39.902528 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0928 02:16:39.903451 (+ 923us) server_negotiation.cc:299] Negotiation successful
0928 02:16:39.903619 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":378,"thread_start_us":234,"threads_started":1}
I20260928 02:16:39.905480 29640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:16:39.884399 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.235.126:33829 (local address 127.0.0.1:41324)
0928 02:16:39.885270 (+ 871us) negotiation.cc:107] Waiting for socket to connect
0928 02:16:39.885317 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0928 02:16:39.887057 (+ 1740us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:16:39.887777 (+ 720us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:16:39.887785 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:16:39.888333 (+ 548us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:16:39.888979 (+ 646us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:16:39.889000 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:16:39.890550 (+ 1550us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:16:39.890554 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:16:39.891864 (+ 1310us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:16:39.891877 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:16:39.893104 (+ 1227us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:16:39.893110 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:16:39.893329 (+ 219us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:16:39.894141 (+ 812us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:16:39.894180 (+ 39us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:16:39.897473 (+ 3293us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:16:39.900731 (+ 3258us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:16:39.900739 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:16:39.900753 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:16:39.901383 (+ 630us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:16:39.901772 (+ 389us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:16:39.901780 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:16:39.901786 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:16:39.902017 (+ 231us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:16:39.902546 (+ 529us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:16:39.902557 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:16:39.903049 (+ 492us) client_negotiation.cc:769] Sending connection context
0928 02:16:39.903360 (+ 311us) client_negotiation.cc:240] Negotiation successful
0928 02:16:39.903935 (+ 575us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":584,"thread_start_us":405,"threads_started":1}
I20260928 02:16:39.907310 29636 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.6.235.125" port: 32979 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 02:16:39 dist-test-slave-86gv krb5kdc[29482](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561799, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.235.125@KRBTEST.COM for kudu/127.6.235.125@KRBTEST.COM
I20260928 02:16:39.923681 29641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:16:39.908381 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59096 (local address 127.6.235.125:32979)
0928 02:16:39.909046 (+ 665us) server_negotiation.cc:206] Beginning negotiation
0928 02:16:39.909052 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 02:16:39.909076 (+ 24us) server_negotiation.cc:407] Connection header received
0928 02:16:39.909119 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:16:39.909133 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:16:39.909311 (+ 178us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:16:39.909458 (+ 147us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:16:39.910203 (+ 745us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:16:39.911248 (+ 1045us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:16:39.912346 (+ 1098us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:16:39.912869 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:16:39.913025 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:16:39.916994 (+ 3969us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:16:39.917006 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:16:39.917021 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:16:39.917056 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:16:39.920297 (+ 3241us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:16:39.921282 (+ 985us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:16:39.921290 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:16:39.921292 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:16:39.921414 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:16:39.922009 (+ 595us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:16:39.922012 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:16:39.922014 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:16:39.922362 (+ 348us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:16:39.922529 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0928 02:16:39.922997 (+ 468us) server_negotiation.cc:299] Negotiation successful
0928 02:16:39.923370 (+ 373us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":170,"server-negotiator.queue_time_us":470,"thread_start_us":348,"threads_started":1}
I20260928 02:16:39.923763 29640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:16:39.908105 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.235.125:32979 (local address 127.0.0.1:59096)
0928 02:16:39.908257 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0928 02:16:39.908272 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 02:16:39.908351 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:16:39.909445 (+ 1094us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:16:39.909450 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:16:39.909885 (+ 435us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:16:39.910013 (+ 128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:16:39.910021 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:16:39.911383 (+ 1362us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:16:39.911388 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:16:39.912215 (+ 827us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:16:39.912226 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:16:39.913050 (+ 824us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:16:39.913059 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:16:39.913311 (+ 252us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:16:39.913913 (+ 602us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:16:39.913939 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:16:39.916695 (+ 2756us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:16:39.920461 (+ 3766us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:16:39.920470 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:16:39.920476 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:16:39.921146 (+ 670us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:16:39.921544 (+ 398us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:16:39.921552 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:16:39.921558 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:16:39.921894 (+ 336us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:16:39.922513 (+ 619us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:16:39.922518 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:16:39.922689 (+ 171us) client_negotiation.cc:769] Sending connection context
0928 02:16:39.922863 (+ 174us) client_negotiation.cc:240] Negotiation successful
0928 02:16:39.923256 (+ 393us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":50}
I20260928 02:16:39.926730 29636 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.6.235.124" port: 34699 } has no permanent_uuid. Determining permanent_uuid...
I20260928 02:16:39.928004 29640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:16:39.927446 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.235.124:34699 (local address 127.0.0.1:39558)
0928 02:16:39.927602 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0928 02:16:39.927829 (+ 227us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.235.124:34699: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
W20260928 02:16:39.928253 29588 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.235.124:34699: connect: Connection refused (error 111)
I20260928 02:16:39.930096 29640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:16:39.929704 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.235.124:34699 (local address 127.0.0.1:39574)
0928 02:16:39.929882 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0928 02:16:39.929966 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.235.124:34699: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
W20260928 02:16:39.930363 29636 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.6.235.124:34699: Network error: Client connection negotiation failed: client connection to 127.6.235.124:34699: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.235.124 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790561432251112-7085-0/krb5kdc/kudu_127.6.235.124.keytab.
Entry for principal kudu/127.6.235.124 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790561432251112-7085-0/krb5kdc/kudu_127.6.235.124.keytab.
I20260928 02:16:39.941314 7085 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTlaUuc/build/asan/bin/kudu
/tmp/dist-test-taskTlaUuc/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790561432251112-7085-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790561432251112-7085-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790561432251112-7085-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790561432251112-7085-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.235.124:34699
--webserver_interface=127.6.235.124
--webserver_port=0
--keytab_file=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790561432251112-7085-0/krb5kdc/kudu_127.6.235.124.keytab
--principal=kudu/127.6.235.124
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.235.84:34081
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.235.126:33829,127.6.235.125:32979,127.6.235.124:34699
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790561432251112-7085-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790561432251112-7085-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790561432251112-7085-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 02:16:39.979738 29636 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.235.124" port: 34699 } attempt: 1
I20260928 02:16:39.981106 29640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:16:39.980546 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.235.124:34699 (local address 127.0.0.1:39582)
0928 02:16:39.980784 (+ 238us) negotiation.cc:107] Waiting for socket to connect
0928 02:16:39.980945 (+ 161us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.235.124:34699: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":109}
I20260928 02:16:39.982213 29640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:16:39.981858 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.235.124:34699 (local address 127.0.0.1:39598)
0928 02:16:39.982024 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0928 02:16:39.982096 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.235.124:34699: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260928 02:16:39.982420 29636 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.6.235.124:34699: Network error: Client connection negotiation failed: client connection to 127.6.235.124:34699: connect: Connection refused (error 111)
I20260928 02:16:40.046959 29562 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.235.125" port: 32979 } attempt: 5
I20260928 02:16:40.050623 29636 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.235.124" port: 34699 } attempt: 2
I20260928 02:16:40.051790 29640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:16:40.051161 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.235.124:34699 (local address 127.0.0.1:39614)
0928 02:16:40.051408 (+ 247us) negotiation.cc:107] Waiting for socket to connect
0928 02:16:40.051537 (+ 129us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.235.124:34699: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":78}
I20260928 02:16:40.052942 29640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:16:40.052489 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.235.124:34699 (local address 127.0.0.1:39628)
0928 02:16:40.052652 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0928 02:16:40.052739 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.235.124:34699: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
W20260928 02:16:40.053323 29636 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.6.235.124:34699: Network error: Client connection negotiation failed: client connection to 127.6.235.124:34699: connect: Connection refused (error 111)
Sep 28 02:16:40 dist-test-slave-86gv krb5kdc[29482](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561799, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.235.126@KRBTEST.COM for kudu/127.6.235.125@KRBTEST.COM
I20260928 02:16:40.061360 29641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:16:40.047876 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59110 (local address 127.6.235.125:32979)
0928 02:16:40.048066 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0928 02:16:40.048072 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 02:16:40.048093 (+ 21us) server_negotiation.cc:407] Connection header received
0928 02:16:40.048308 (+ 215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:16:40.048314 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:16:40.048377 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:16:40.048485 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:16:40.049328 (+ 843us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:16:40.050024 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:16:40.051498 (+ 1474us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:16:40.052187 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:16:40.052631 (+ 444us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:16:40.055699 (+ 3068us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:16:40.055704 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:16:40.055707 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:16:40.055740 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:16:40.058689 (+ 2949us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:16:40.059360 (+ 671us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:16:40.059367 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:16:40.059370 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:16:40.059510 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:16:40.059899 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:16:40.059905 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:16:40.059908 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:16:40.060199 (+ 291us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:16:40.060355 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0928 02:16:40.060738 (+ 383us) server_negotiation.cc:299] Negotiation successful
0928 02:16:40.060988 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260928 02:16:40.061364 29564 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:16:40.047759 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.235.125:32979 (local address 127.0.0.1:59110)
0928 02:16:40.047989 (+ 230us) negotiation.cc:107] Waiting for socket to connect
0928 02:16:40.048009 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0928 02:16:40.048142 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:16:40.048518 (+ 376us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:16:40.048523 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:16:40.048996 (+ 473us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:16:40.049175 (+ 179us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:16:40.049184 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:16:40.050168 (+ 984us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:16:40.050175 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:16:40.051354 (+ 1179us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:16:40.051366 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:16:40.052339 (+ 973us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:16:40.052347 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:16:40.052587 (+ 240us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:16:40.053368 (+ 781us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:16:40.053393 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:16:40.055489 (+ 2096us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:16:40.058829 (+ 3340us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:16:40.058835 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:16:40.058839 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:16:40.059226 (+ 387us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:16:40.059637 (+ 411us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:16:40.059642 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:16:40.059646 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:16:40.059788 (+ 142us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:16:40.060322 (+ 534us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:16:40.060328 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:16:40.060489 (+ 161us) client_negotiation.cc:769] Sending connection context
0928 02:16:40.060641 (+ 152us) client_negotiation.cc:240] Negotiation successful
0928 02:16:40.060987 (+ 346us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":80}
I20260928 02:16:40.062382 29562 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.6.235.124" port: 34699 } has no permanent_uuid. Determining permanent_uuid...
I20260928 02:16:40.063472 29564 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:16:40.063021 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.235.124:34699 (local address 127.0.0.1:39638)
0928 02:16:40.063204 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0928 02:16:40.063301 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.235.124:34699: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
I20260928 02:16:40.064554 29564 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:16:40.064111 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.235.124:34699 (local address 127.0.0.1:39652)
0928 02:16:40.064293 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0928 02:16:40.064406 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.235.124:34699: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260928 02:16:40.064796 29562 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.6.235.124:34699: Network error: Client connection negotiation failed: client connection to 127.6.235.124:34699: connect: Connection refused (error 111)
I20260928 02:16:40.116106 29562 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.235.124" port: 34699 } attempt: 1
I20260928 02:16:40.117588 29564 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:16:40.116944 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.235.124:34699 (local address 127.0.0.1:39654)
0928 02:16:40.117205 (+ 261us) negotiation.cc:107] Waiting for socket to connect
0928 02:16:40.117340 (+ 135us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.235.124:34699: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":78}
I20260928 02:16:40.118918 29564 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:16:40.118443 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.235.124:34699 (local address 127.0.0.1:39660)
0928 02:16:40.118639 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0928 02:16:40.118749 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.235.124:34699: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
W20260928 02:16:40.119248 29562 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.6.235.124:34699: Network error: Client connection negotiation failed: client connection to 127.6.235.124:34699: connect: Connection refused (error 111)
I20260928 02:16:40.144570 29636 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.235.124" port: 34699 } attempt: 3
I20260928 02:16:40.145915 29640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:16:40.145382 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.235.124:34699 (local address 127.0.0.1:39676)
0928 02:16:40.145595 (+ 213us) negotiation.cc:107] Waiting for socket to connect
0928 02:16:40.145759 (+ 164us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.235.124:34699: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":72}
I20260928 02:16:40.147001 29640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:16:40.146622 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.235.124:34699 (local address 127.0.0.1:39680)
0928 02:16:40.146829 (+ 207us) negotiation.cc:107] Waiting for socket to connect
0928 02:16:40.146899 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.235.124:34699: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":131}
W20260928 02:16:40.147243 29636 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.6.235.124:34699: Network error: Client connection negotiation failed: client connection to 127.6.235.124:34699: connect: Connection refused (error 111)
I20260928 02:16:40.187525 29562 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.235.124" port: 34699 } attempt: 2
I20260928 02:16:40.188879 29564 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:16:40.188248 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.235.124:34699 (local address 127.0.0.1:39692)
0928 02:16:40.188533 (+ 285us) negotiation.cc:107] Waiting for socket to connect
0928 02:16:40.188660 (+ 127us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.235.124:34699: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":81}
I20260928 02:16:40.190181 29564 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:16:40.189763 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.235.124:34699 (local address 127.0.0.1:39700)
0928 02:16:40.189925 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0928 02:16:40.190043 (+ 118us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.235.124:34699: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260928 02:16:40.190593 29562 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.6.235.124:34699: Network error: Client connection negotiation failed: client connection to 127.6.235.124:34699: connect: Connection refused (error 111)
I20260928 02:16:40.290490 29636 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.235.124" port: 34699 } attempt: 4
I20260928 02:16:40.291848 29640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:16:40.291283 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.235.124:34699 (local address 127.0.0.1:39706)
0928 02:16:40.291519 (+ 236us) negotiation.cc:107] Waiting for socket to connect
0928 02:16:40.291684 (+ 165us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.235.124:34699: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":77}
I20260928 02:16:40.292852 29640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:16:40.292544 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.235.124:34699 (local address 127.0.0.1:39714)
0928 02:16:40.292688 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0928 02:16:40.292759 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.235.124:34699: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260928 02:16:40.293089 29636 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.6.235.124:34699: Network error: Client connection negotiation failed: client connection to 127.6.235.124:34699: connect: Connection refused (error 111)
I20260928 02:16:40.296880 29562 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.235.124" port: 34699 } attempt: 3
W20260928 02:16:40.297451 29644 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260928 02:16:40.297834 29644 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:16:40.297961 29644 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:16:40.298107 29644 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260928 02:16:40.298280 29564 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:16:40.297693 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.235.124:34699 (local address 127.0.0.1:39726)
0928 02:16:40.297899 (+ 206us) negotiation.cc:107] Waiting for socket to connect
0928 02:16:40.298056 (+ 157us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.235.124:34699: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":69}
I20260928 02:16:40.299446 29564 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:16:40.298979 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.235.124:34699 (local address 127.0.0.1:39728)
0928 02:16:40.299244 (+ 265us) negotiation.cc:107] Waiting for socket to connect
0928 02:16:40.299344 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.235.124:34699: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":144}
W20260928 02:16:40.299666 29562 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.6.235.124:34699: Network error: Client connection negotiation failed: client connection to 127.6.235.124:34699: connect: Connection refused (error 111)
W20260928 02:16:40.306192 29644 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 02:16:40.306279 29644 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:16:40.306376 29644 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:16:40.306442 29644 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 02:16:40.306615 29644 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 02:16:40.316774 29644 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.235.84:34081
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790561432251112-7085-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790561432251112-7085-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.6.235.126:33829,127.6.235.125:32979,127.6.235.124:34699
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790561432251112-7085-0/krb5kdc/kudu_127.6.235.124.keytab
--principal=kudu/127.6.235.124
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.235.124:34699
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790561432251112-7085-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.6.235.124
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790561432251112-7085-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 00:51:59 UTC on ccycloud.quasar-axgkwn.root.comops.site
build id 4676464
ASAN enabled
I20260928 02:16:40.318215 29644 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:16:40.319633 29644 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:16:40.333570 29650 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:16:40.333657 29652 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:16:40.333657 29649 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:16:40.334676 29644 server_base.cc:1034] running on GCE node
I20260928 02:16:40.335498 29644 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:16:40.337160 29644 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:16:40.338441 29644 hybrid_clock.cc:648] HybridClock initialized: now 1790561800338367 us; error 52 us; skew 500 ppm
I20260928 02:16:40.338842 29644 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 02:16:40 dist-test-slave-86gv krb5kdc[29482](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561800, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.235.124@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:16:40.342567 29644 init.cc:377] Logged in from keytab as kudu/127.6.235.124@KRBTEST.COM (short username kudu)
I20260928 02:16:40.344213 29644 webserver.cc:460] Webserver started at http://127.6.235.124:45865/ using document root <none> and password file <none>
I20260928 02:16:40.345036 29644 fs_manager.cc:359] Metadata directory not provided
I20260928 02:16:40.345177 29644 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:16:40.345499 29644 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:16:40.348539 29644 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790561432251112-7085-0/minicluster-data/master-2/data/instance:
uuid: "bbc1fb6d83d04546aa8a14b43e8e8084"
format_stamp: "Formatted at 2026-09-28 02:16:40 on dist-test-slave-86gv"
server_key: "07afedf2fa3d0a6d22189c0a09b66d1a"
server_key_iv: "366b71c3f297a567756e13efc4ecb891"
server_key_version: "encryptionkey@0"
I20260928 02:16:40.349411 29644 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790561432251112-7085-0/minicluster-data/master-2/wal/instance:
uuid: "bbc1fb6d83d04546aa8a14b43e8e8084"
format_stamp: "Formatted at 2026-09-28 02:16:40 on dist-test-slave-86gv"
server_key: "07afedf2fa3d0a6d22189c0a09b66d1a"
server_key_iv: "366b71c3f297a567756e13efc4ecb891"
server_key_version: "encryptionkey@0"
I20260928 02:16:40.355067 29644 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260928 02:16:40.358891 29659 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:16:40.360445 29644 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260928 02:16:40.360666 29644 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790561432251112-7085-0/minicluster-data/master-2/data,/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790561432251112-7085-0/minicluster-data/master-2/wal
uuid: "bbc1fb6d83d04546aa8a14b43e8e8084"
format_stamp: "Formatted at 2026-09-28 02:16:40 on dist-test-slave-86gv"
server_key: "07afedf2fa3d0a6d22189c0a09b66d1a"
server_key_iv: "366b71c3f297a567756e13efc4ecb891"
server_key_version: "encryptionkey@0"
I20260928 02:16:40.360900 29644 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790561432251112-7085-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790561432251112-7085-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790561432251112-7085-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:16:40.384594 29644 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:16:40.389750 29644 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:16:40.390061 29644 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:16:40.410565 29644 rpc_server.cc:307] RPC server started. Bound to: 127.6.235.124:34699
I20260928 02:16:40.410589 29710 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.235.124:34699 every 8 connection(s)
I20260928 02:16:40.412073 29644 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790561432251112-7085-0/minicluster-data/master-2/data/info.pb
I20260928 02:16:40.416332 29711 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:16:40.420866 7085 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTlaUuc/build/asan/bin/kudu as pid 29644
I20260928 02:16:40.421156 7085 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790561432251112-7085-0/minicluster-data/master-2/wal/instance
I20260928 02:16:40.421692 7085 external_mini_cluster.cc:1428] Setting key 2d85c7d8d01720470832b620239c4730
I20260928 02:16:40.424113 29711 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.6.235.126" port: 33829 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 02:16:40 dist-test-slave-86gv krb5kdc[29482](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561800, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.235.124@KRBTEST.COM for kudu/127.6.235.126@KRBTEST.COM
Sep 28 02:16:40 dist-test-slave-86gv krb5kdc[29482](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561798, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.235.124@KRBTEST.COM
I20260928 02:16:40.444473 29715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:16:40.427016 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41330 (local address 127.6.235.126:33829)
0928 02:16:40.427618 (+ 602us) server_negotiation.cc:206] Beginning negotiation
0928 02:16:40.427623 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 02:16:40.428157 (+ 534us) server_negotiation.cc:407] Connection header received
0928 02:16:40.428428 (+ 271us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:16:40.428432 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:16:40.428520 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:16:40.428659 (+ 139us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:16:40.430163 (+ 1504us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:16:40.431236 (+ 1073us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:16:40.432877 (+ 1641us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:16:40.433625 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:16:40.433763 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:16:40.437688 (+ 3925us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:16:40.437695 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:16:40.437701 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:16:40.437777 (+ 76us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:16:40.440598 (+ 2821us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:16:40.441571 (+ 973us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:16:40.441574 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:16:40.441576 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:16:40.441744 (+ 168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:16:40.442280 (+ 536us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:16:40.442288 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:16:40.442291 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:16:40.442760 (+ 469us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:16:40.442959 (+ 199us) server_negotiation.cc:1035] Waiting for connection context
0928 02:16:40.443907 (+ 948us) server_negotiation.cc:299] Negotiation successful
0928 02:16:40.444078 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":456,"thread_start_us":310,"threads_started":1}
I20260928 02:16:40.445853 29714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:16:40.423863 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39734 (local address 127.6.235.124:34699)
0928 02:16:40.424746 (+ 883us) server_negotiation.cc:206] Beginning negotiation
0928 02:16:40.424758 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0928 02:16:40.424802 (+ 44us) server_negotiation.cc:407] Connection header received
0928 02:16:40.426593 (+ 1791us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:16:40.426617 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:16:40.427115 (+ 498us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:16:40.428402 (+ 1287us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:16:40.429447 (+ 1045us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:16:40.431236 (+ 1789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:16:40.432552 (+ 1316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:16:40.433439 (+ 887us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:16:40.433649 (+ 210us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:16:40.438287 (+ 4638us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:16:40.438295 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:16:40.438309 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:16:40.438340 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:16:40.441992 (+ 3652us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:16:40.442933 (+ 941us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:16:40.442943 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:16:40.442947 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:16:40.443135 (+ 188us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:16:40.443632 (+ 497us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:16:40.443638 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:16:40.443641 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:16:40.444102 (+ 461us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:16:40.444430 (+ 328us) server_negotiation.cc:1035] Waiting for connection context
0928 02:16:40.445146 (+ 716us) server_negotiation.cc:299] Negotiation successful
0928 02:16:40.445375 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":662,"spinlock_wait_cycles":27008,"thread_start_us":438,"threads_started":1}
I20260928 02:16:40.445931 29716 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:16:40.427080 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.235.126:33829 (local address 127.0.0.1:41330)
0928 02:16:40.428004 (+ 924us) negotiation.cc:107] Waiting for socket to connect
0928 02:16:40.428051 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0928 02:16:40.428222 (+ 171us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:16:40.428862 (+ 640us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:16:40.428867 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:16:40.429589 (+ 722us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:16:40.429971 (+ 382us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:16:40.429988 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:16:40.431442 (+ 1454us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:16:40.431446 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:16:40.432671 (+ 1225us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:16:40.432683 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:16:40.434023 (+ 1340us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:16:40.434030 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:16:40.434166 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:16:40.434889 (+ 723us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:16:40.434915 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:16:40.437439 (+ 2524us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:16:40.440745 (+ 3306us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:16:40.440756 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:16:40.440773 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:16:40.441459 (+ 686us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:16:40.441857 (+ 398us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:16:40.441866 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:16:40.441870 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:16:40.442122 (+ 252us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:16:40.442933 (+ 811us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:16:40.442943 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:16:40.443429 (+ 486us) client_negotiation.cc:769] Sending connection context
0928 02:16:40.443799 (+ 370us) client_negotiation.cc:240] Negotiation successful
0928 02:16:40.444414 (+ 615us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":683,"thread_start_us":539,"threads_started":1}
I20260928 02:16:40.447533 29711 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.6.235.125" port: 32979 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 02:16:40 dist-test-slave-86gv krb5kdc[29482](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561800, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.235.124@KRBTEST.COM for kudu/127.6.235.125@KRBTEST.COM
Sep 28 02:16:40 dist-test-slave-86gv krb5kdc[29482](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561798, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.235.126@KRBTEST.COM
I20260928 02:16:40.462775 29716 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:16:40.448346 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.235.125:32979 (local address 127.0.0.1:59124)
0928 02:16:40.448608 (+ 262us) negotiation.cc:107] Waiting for socket to connect
0928 02:16:40.448630 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0928 02:16:40.448717 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:16:40.449088 (+ 371us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:16:40.449094 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:16:40.449488 (+ 394us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:16:40.449670 (+ 182us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:16:40.449679 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:16:40.450783 (+ 1104us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:16:40.450788 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:16:40.451777 (+ 989us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:16:40.451789 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:16:40.452607 (+ 818us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:16:40.452614 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:16:40.452862 (+ 248us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:16:40.453555 (+ 693us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:16:40.453572 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:16:40.456087 (+ 2515us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:16:40.459761 (+ 3674us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:16:40.459766 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:16:40.459769 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:16:40.460332 (+ 563us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:16:40.460756 (+ 424us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:16:40.460760 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:16:40.460762 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:16:40.460997 (+ 235us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:16:40.461761 (+ 764us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:16:40.461765 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:16:40.461907 (+ 142us) client_negotiation.cc:769] Sending connection context
0928 02:16:40.462031 (+ 124us) client_negotiation.cc:240] Negotiation successful
0928 02:16:40.462266 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":161}
I20260928 02:16:40.462802 29641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:16:40.448358 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59124 (local address 127.6.235.125:32979)
0928 02:16:40.448588 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0928 02:16:40.448595 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 02:16:40.448704 (+ 109us) server_negotiation.cc:407] Connection header received
0928 02:16:40.448810 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:16:40.448815 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:16:40.448930 (+ 115us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:16:40.449061 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:16:40.449796 (+ 735us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:16:40.450636 (+ 840us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:16:40.451935 (+ 1299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:16:40.452460 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:16:40.452561 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:16:40.456283 (+ 3722us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:16:40.456289 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:16:40.456292 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:16:40.456326 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:16:40.459589 (+ 3263us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:16:40.460459 (+ 870us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:16:40.460465 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:16:40.460469 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:16:40.460636 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:16:40.461128 (+ 492us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:16:40.461133 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:16:40.461137 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:16:40.461554 (+ 417us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:16:40.461793 (+ 239us) server_negotiation.cc:1035] Waiting for connection context
0928 02:16:40.462152 (+ 359us) server_negotiation.cc:299] Negotiation successful
0928 02:16:40.462383 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260928 02:16:40.464102 29711 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.6.235.124" port: 34699 } has no permanent_uuid. Determining permanent_uuid...
I20260928 02:16:40.466320 29715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:16:40.449268 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41342 (local address 127.6.235.126:33829)
0928 02:16:40.449449 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0928 02:16:40.449456 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 02:16:40.449475 (+ 19us) server_negotiation.cc:407] Connection header received
0928 02:16:40.449714 (+ 239us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:16:40.449720 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:16:40.449813 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:16:40.449923 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:16:40.451111 (+ 1188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:16:40.451912 (+ 801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:16:40.453369 (+ 1457us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:16:40.453932 (+ 563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:16:40.454042 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:16:40.459225 (+ 5183us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:16:40.459232 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:16:40.459237 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:16:40.459283 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:16:40.462775 (+ 3492us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:16:40.463546 (+ 771us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:16:40.463552 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:16:40.463556 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:16:40.463701 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:16:40.464084 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:16:40.464089 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:16:40.464092 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:16:40.464437 (+ 345us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:16:40.464776 (+ 339us) server_negotiation.cc:1035] Waiting for connection context
0928 02:16:40.465735 (+ 959us) server_negotiation.cc:299] Negotiation successful
0928 02:16:40.465975 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
Sep 28 02:16:40 dist-test-slave-86gv krb5kdc[29482](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561800, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.235.124@KRBTEST.COM for kudu/127.6.235.124@KRBTEST.COM
I20260928 02:16:40.476969 29562 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.235.124" port: 34699 } attempt: 4
Sep 28 02:16:40 dist-test-slave-86gv krb5kdc[29482](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561798, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.235.125@KRBTEST.COM
I20260928 02:16:40.480098 29716 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:16:40.464871 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.235.124:34699 (local address 127.0.0.1:39746)
0928 02:16:40.465083 (+ 212us) negotiation.cc:107] Waiting for socket to connect
0928 02:16:40.465102 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0928 02:16:40.465203 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:16:40.465481 (+ 278us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:16:40.465485 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:16:40.465967 (+ 482us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:16:40.466161 (+ 194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:16:40.466172 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:16:40.467604 (+ 1432us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:16:40.467611 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:16:40.468991 (+ 1380us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:16:40.469009 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:16:40.469733 (+ 724us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:16:40.469739 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:16:40.469987 (+ 248us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:16:40.470707 (+ 720us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:16:40.470723 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:16:40.473231 (+ 2508us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:16:40.477270 (+ 4039us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:16:40.477275 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:16:40.477280 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:16:40.477861 (+ 581us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:16:40.478284 (+ 423us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:16:40.478290 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:16:40.478293 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:16:40.478431 (+ 138us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:16:40.478952 (+ 521us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:16:40.478957 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:16:40.479101 (+ 144us) client_negotiation.cc:769] Sending connection context
0928 02:16:40.479245 (+ 144us) client_negotiation.cc:240] Negotiation successful
0928 02:16:40.479661 (+ 416us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":89}
I20260928 02:16:40.480489 29714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:16:40.464886 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39746 (local address 127.6.235.124:34699)
0928 02:16:40.465093 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0928 02:16:40.465099 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 02:16:40.465164 (+ 65us) server_negotiation.cc:407] Connection header received
0928 02:16:40.465290 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:16:40.465294 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:16:40.465352 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:16:40.465431 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:16:40.466327 (+ 896us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:16:40.467472 (+ 1145us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:16:40.469137 (+ 1665us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:16:40.469574 (+ 437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:16:40.469713 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:16:40.473449 (+ 3736us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:16:40.473455 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:16:40.473459 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:16:40.473496 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:16:40.477086 (+ 3590us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:16:40.477983 (+ 897us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:16:40.477987 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:16:40.477990 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:16:40.478162 (+ 172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:16:40.478542 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:16:40.478546 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:16:40.478549 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:16:40.478836 (+ 287us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:16:40.478991 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0928 02:16:40.479811 (+ 820us) server_negotiation.cc:299] Negotiation successful
0928 02:16:40.480082 (+ 271us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260928 02:16:40.486509 29711 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bbc1fb6d83d04546aa8a14b43e8e8084: Bootstrap starting.
I20260928 02:16:40.487021 29641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:16:40.468073 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59130 (local address 127.6.235.125:32979)
0928 02:16:40.468240 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0928 02:16:40.468245 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 02:16:40.468261 (+ 16us) server_negotiation.cc:407] Connection header received
0928 02:16:40.468381 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:16:40.468385 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:16:40.468443 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:16:40.468524 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:16:40.470047 (+ 1523us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:16:40.471085 (+ 1038us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:16:40.472583 (+ 1498us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:16:40.473327 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:16:40.473449 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:16:40.479337 (+ 5888us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:16:40.479346 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:16:40.479351 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:16:40.479399 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:16:40.483792 (+ 4393us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:16:40.484668 (+ 876us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:16:40.484674 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:16:40.484677 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:16:40.484840 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:16:40.485325 (+ 485us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:16:40.485331 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:16:40.485336 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:16:40.485728 (+ 392us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:16:40.485993 (+ 265us) server_negotiation.cc:1035] Waiting for connection context
0928 02:16:40.486298 (+ 305us) server_negotiation.cc:299] Negotiation successful
0928 02:16:40.486486 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
Sep 28 02:16:40 dist-test-slave-86gv krb5kdc[29482](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561799, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.235.126@KRBTEST.COM for kudu/127.6.235.124@KRBTEST.COM
I20260928 02:16:40.490836 29711 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bbc1fb6d83d04546aa8a14b43e8e8084: Neither blocks nor log segments found. Creating new log.
I20260928 02:16:40.492388 29711 log.cc:824] T 00000000000000000000000000000000 P bbc1fb6d83d04546aa8a14b43e8e8084: Log is configured to *not* fsync() on all Append() calls
I20260928 02:16:40.492552 7085 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 02:16:40.496018 29711 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bbc1fb6d83d04546aa8a14b43e8e8084: No bootstrap required, opened a new log
I20260928 02:16:40.497507 29564 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:16:40.477694 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.235.124:34699 (local address 127.0.0.1:39762)
0928 02:16:40.477987 (+ 293us) negotiation.cc:107] Waiting for socket to connect
0928 02:16:40.478006 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0928 02:16:40.478115 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:16:40.480528 (+ 2413us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:16:40.480534 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:16:40.481104 (+ 570us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:16:40.481314 (+ 210us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:16:40.481330 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:16:40.482640 (+ 1310us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:16:40.482645 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:16:40.484079 (+ 1434us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:16:40.484095 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:16:40.485116 (+ 1021us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:16:40.485122 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:16:40.485307 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:16:40.486848 (+ 1541us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:16:40.486877 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:16:40.490071 (+ 3194us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:16:40.494226 (+ 4155us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:16:40.494233 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:16:40.494239 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:16:40.494828 (+ 589us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:16:40.495292 (+ 464us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:16:40.495299 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:16:40.495303 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:16:40.495511 (+ 208us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:16:40.496289 (+ 778us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:16:40.496296 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:16:40.496494 (+ 198us) client_negotiation.cc:769] Sending connection context
0928 02:16:40.496729 (+ 235us) client_negotiation.cc:240] Negotiation successful
0928 02:16:40.497096 (+ 367us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":163}
I20260928 02:16:40.498004 29717 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:16:40.477831 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39762 (local address 127.6.235.124:34699)
0928 02:16:40.479884 (+ 2053us) server_negotiation.cc:206] Beginning negotiation
0928 02:16:40.479890 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 02:16:40.479911 (+ 21us) server_negotiation.cc:407] Connection header received
0928 02:16:40.479960 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:16:40.480277 (+ 317us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:16:40.480366 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:16:40.480493 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:16:40.481502 (+ 1009us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:16:40.482486 (+ 984us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:16:40.484243 (+ 1757us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:16:40.484957 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:16:40.485102 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:16:40.490323 (+ 5221us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:16:40.490330 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:16:40.490338 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:16:40.490377 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:16:40.494033 (+ 3656us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:16:40.494966 (+ 933us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:16:40.494974 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:16:40.494979 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:16:40.495149 (+ 170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:16:40.495658 (+ 509us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:16:40.495664 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:16:40.495667 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:16:40.496050 (+ 383us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:16:40.496195 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0928 02:16:40.496860 (+ 665us) server_negotiation.cc:299] Negotiation successful
0928 02:16:40.497142 (+ 282us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1907,"thread_start_us":1656,"threads_started":1}
I20260928 02:16:40.496405 29537 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "f48dda22055547a596b2e2575946f3a4"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41342
I20260928 02:16:40.504316 29711 raft_consensus.cc:348] T 00000000000000000000000000000000 P bbc1fb6d83d04546aa8a14b43e8e8084 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f48dda22055547a596b2e2575946f3a4" member_type: VOTER last_known_addr { host: "127.6.235.126" port: 33829 } } peers { permanent_uuid: "aab697abb3f64c20a42d6b6c684ea39f" member_type: VOTER last_known_addr { host: "127.6.235.125" port: 32979 } } peers { permanent_uuid: "bbc1fb6d83d04546aa8a14b43e8e8084" member_type: VOTER last_known_addr { host: "127.6.235.124" port: 34699 } }
I20260928 02:16:40.504657 29711 raft_consensus.cc:729] T 00000000000000000000000000000000 P bbc1fb6d83d04546aa8a14b43e8e8084 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bbc1fb6d83d04546aa8a14b43e8e8084, State: Initialized, Role: FOLLOWER
I20260928 02:16:40.504920 29562 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f48dda22055547a596b2e2575946f3a4: Bootstrap starting.
I20260928 02:16:40.505174 29711 consensus_queue.cc:260] T 00000000000000000000000000000000 P bbc1fb6d83d04546aa8a14b43e8e8084 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f48dda22055547a596b2e2575946f3a4" member_type: VOTER last_known_addr { host: "127.6.235.126" port: 33829 } } peers { permanent_uuid: "aab697abb3f64c20a42d6b6c684ea39f" member_type: VOTER last_known_addr { host: "127.6.235.125" port: 32979 } } peers { permanent_uuid: "bbc1fb6d83d04546aa8a14b43e8e8084" member_type: VOTER last_known_addr { host: "127.6.235.124" port: 34699 } }
I20260928 02:16:40.506152 29719 sys_catalog.cc:455] T 00000000000000000000000000000000 P bbc1fb6d83d04546aa8a14b43e8e8084 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f48dda22055547a596b2e2575946f3a4" member_type: VOTER last_known_addr { host: "127.6.235.126" port: 33829 } } peers { permanent_uuid: "aab697abb3f64c20a42d6b6c684ea39f" member_type: VOTER last_known_addr { host: "127.6.235.125" port: 32979 } } peers { permanent_uuid: "bbc1fb6d83d04546aa8a14b43e8e8084" member_type: VOTER last_known_addr { host: "127.6.235.124" port: 34699 } } }
I20260928 02:16:40.506626 29719 sys_catalog.cc:458] T 00000000000000000000000000000000 P bbc1fb6d83d04546aa8a14b43e8e8084 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 02:16:40.507475 29711 sys_catalog.cc:564] T 00000000000000000000000000000000 P bbc1fb6d83d04546aa8a14b43e8e8084 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 02:16:40.509799 29562 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f48dda22055547a596b2e2575946f3a4: Neither blocks nor log segments found. Creating new log.
I20260928 02:16:40.511315 29562 log.cc:824] T 00000000000000000000000000000000 P f48dda22055547a596b2e2575946f3a4: Log is configured to *not* fsync() on all Append() calls
I20260928 02:16:40.514603 29562 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f48dda22055547a596b2e2575946f3a4: No bootstrap required, opened a new log
I20260928 02:16:40.522663 29562 raft_consensus.cc:348] T 00000000000000000000000000000000 P f48dda22055547a596b2e2575946f3a4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f48dda22055547a596b2e2575946f3a4" member_type: VOTER last_known_addr { host: "127.6.235.126" port: 33829 } } peers { permanent_uuid: "aab697abb3f64c20a42d6b6c684ea39f" member_type: VOTER last_known_addr { host: "127.6.235.125" port: 32979 } } peers { permanent_uuid: "bbc1fb6d83d04546aa8a14b43e8e8084" member_type: VOTER last_known_addr { host: "127.6.235.124" port: 34699 } }
I20260928 02:16:40.523134 29562 raft_consensus.cc:729] T 00000000000000000000000000000000 P f48dda22055547a596b2e2575946f3a4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f48dda22055547a596b2e2575946f3a4, State: Initialized, Role: FOLLOWER
I20260928 02:16:40.523725 29562 consensus_queue.cc:260] T 00000000000000000000000000000000 P f48dda22055547a596b2e2575946f3a4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f48dda22055547a596b2e2575946f3a4" member_type: VOTER last_known_addr { host: "127.6.235.126" port: 33829 } } peers { permanent_uuid: "aab697abb3f64c20a42d6b6c684ea39f" member_type: VOTER last_known_addr { host: "127.6.235.125" port: 32979 } } peers { permanent_uuid: "bbc1fb6d83d04546aa8a14b43e8e8084" member_type: VOTER last_known_addr { host: "127.6.235.124" port: 34699 } }
W20260928 02:16:40.524669 29731 catalog_manager.cc:1558] T 00000000000000000000000000000000 P bbc1fb6d83d04546aa8a14b43e8e8084: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 02:16:40.524857 29731 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 02:16:40.525977 29733 sys_catalog.cc:455] T 00000000000000000000000000000000 P f48dda22055547a596b2e2575946f3a4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f48dda22055547a596b2e2575946f3a4" member_type: VOTER last_known_addr { host: "127.6.235.126" port: 33829 } } peers { permanent_uuid: "aab697abb3f64c20a42d6b6c684ea39f" member_type: VOTER last_known_addr { host: "127.6.235.125" port: 32979 } } peers { permanent_uuid: "bbc1fb6d83d04546aa8a14b43e8e8084" member_type: VOTER last_known_addr { host: "127.6.235.124" port: 34699 } } }
I20260928 02:16:40.526726 29733 sys_catalog.cc:458] T 00000000000000000000000000000000 P f48dda22055547a596b2e2575946f3a4 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 02:16:40.527251 29562 sys_catalog.cc:564] T 00000000000000000000000000000000 P f48dda22055547a596b2e2575946f3a4 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 02:16:40.527405 29537 raft_consensus.cc:482] T 00000000000000000000000000000000 P f48dda22055547a596b2e2575946f3a4 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260928 02:16:40.527628 29537 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f48dda22055547a596b2e2575946f3a4 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:16:40.531095 29537 raft_consensus.cc:504] T 00000000000000000000000000000000 P f48dda22055547a596b2e2575946f3a4 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f48dda22055547a596b2e2575946f3a4" member_type: VOTER last_known_addr { host: "127.6.235.126" port: 33829 } } peers { permanent_uuid: "aab697abb3f64c20a42d6b6c684ea39f" member_type: VOTER last_known_addr { host: "127.6.235.125" port: 32979 } } peers { permanent_uuid: "bbc1fb6d83d04546aa8a14b43e8e8084" member_type: VOTER last_known_addr { host: "127.6.235.124" port: 34699 } }
I20260928 02:16:40.533754 29537 leader_election.cc:288] T 00000000000000000000000000000000 P f48dda22055547a596b2e2575946f3a4 [CANDIDATE]: Term 1 election: Requested vote from peers aab697abb3f64c20a42d6b6c684ea39f (127.6.235.125:32979), bbc1fb6d83d04546aa8a14b43e8e8084 (127.6.235.124:34699)
I20260928 02:16:40.534525 29686 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "f48dda22055547a596b2e2575946f3a4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "bbc1fb6d83d04546aa8a14b43e8e8084"
I20260928 02:16:40.535099 29686 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bbc1fb6d83d04546aa8a14b43e8e8084 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:16:40.522305 29644 master_runner.cc:186] Error getting master registration for 127.6.235.126:33829: OK, instance_id { permanent_uuid: "f48dda22055547a596b2e2575946f3a4" instance_seqno: 1790561799331217 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260928 02:16:40.536839 29644 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260928 02:16:40.538704 29686 raft_consensus.cc:2454] T 00000000000000000000000000000000 P bbc1fb6d83d04546aa8a14b43e8e8084 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f48dda22055547a596b2e2575946f3a4 in term 1.
I20260928 02:16:40.539466 29514 leader_election.cc:302] T 00000000000000000000000000000000 P f48dda22055547a596b2e2575946f3a4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bbc1fb6d83d04546aa8a14b43e8e8084, f48dda22055547a596b2e2575946f3a4; no voters:
I20260928 02:16:40.540159 29733 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f48dda22055547a596b2e2575946f3a4 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:16:40.547973 29733 raft_consensus.cc:686] T 00000000000000000000000000000000 P f48dda22055547a596b2e2575946f3a4 [term 1 LEADER]: Becoming Leader. State: Replica: f48dda22055547a596b2e2575946f3a4, State: Running, Role: LEADER
I20260928 02:16:40.548883 29733 consensus_queue.cc:237] T 00000000000000000000000000000000 P f48dda22055547a596b2e2575946f3a4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f48dda22055547a596b2e2575946f3a4" member_type: VOTER last_known_addr { host: "127.6.235.126" port: 33829 } } peers { permanent_uuid: "aab697abb3f64c20a42d6b6c684ea39f" member_type: VOTER last_known_addr { host: "127.6.235.125" port: 32979 } } peers { permanent_uuid: "bbc1fb6d83d04546aa8a14b43e8e8084" member_type: VOTER last_known_addr { host: "127.6.235.124" port: 34699 } }
I20260928 02:16:40.558197 29735 sys_catalog.cc:455] T 00000000000000000000000000000000 P f48dda22055547a596b2e2575946f3a4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f48dda22055547a596b2e2575946f3a4. Latest consensus state: current_term: 1 leader_uuid: "f48dda22055547a596b2e2575946f3a4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f48dda22055547a596b2e2575946f3a4" member_type: VOTER last_known_addr { host: "127.6.235.126" port: 33829 } } peers { permanent_uuid: "aab697abb3f64c20a42d6b6c684ea39f" member_type: VOTER last_known_addr { host: "127.6.235.125" port: 32979 } } peers { permanent_uuid: "bbc1fb6d83d04546aa8a14b43e8e8084" member_type: VOTER last_known_addr { host: "127.6.235.124" port: 34699 } } }
I20260928 02:16:40.558866 29735 sys_catalog.cc:458] T 00000000000000000000000000000000 P f48dda22055547a596b2e2575946f3a4 [sys.catalog]: This master's current role is: LEADER
W20260928 02:16:40.570396 29516 tablet.cc:2369] T 00000000000000000000000000000000 P f48dda22055547a596b2e2575946f3a4: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 02:16:40.580840 29744 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 02:16:40.592320 29636 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.235.124" port: 34699 } attempt: 5
Sep 28 02:16:40 dist-test-slave-86gv krb5kdc[29482](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561799, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.235.125@KRBTEST.COM for kudu/127.6.235.124@KRBTEST.COM
I20260928 02:16:40.615371 29714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:16:40.593551 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39780 (local address 127.6.235.124:34699)
0928 02:16:40.593762 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0928 02:16:40.593768 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 02:16:40.593792 (+ 24us) server_negotiation.cc:407] Connection header received
0928 02:16:40.593848 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:16:40.593856 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:16:40.593938 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:16:40.594046 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:16:40.595281 (+ 1235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:16:40.596617 (+ 1336us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:16:40.598634 (+ 2017us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:16:40.599556 (+ 922us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:16:40.599714 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:16:40.604908 (+ 5194us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:16:40.604918 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:16:40.604926 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:16:40.604973 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:16:40.609669 (+ 4696us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:16:40.611234 (+ 1565us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:16:40.611240 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:16:40.611259 (+ 19us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:16:40.611431 (+ 172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:16:40.612277 (+ 846us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:16:40.612284 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:16:40.612288 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:16:40.612718 (+ 430us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:16:40.612867 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0928 02:16:40.613989 (+ 1122us) server_negotiation.cc:299] Negotiation successful
0928 02:16:40.614530 (+ 541us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260928 02:16:40.617388 29640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:16:40.593085 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.235.124:34699 (local address 127.0.0.1:39780)
0928 02:16:40.593327 (+ 242us) negotiation.cc:107] Waiting for socket to connect
0928 02:16:40.593356 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0928 02:16:40.593506 (+ 150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:16:40.594088 (+ 582us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:16:40.594096 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:16:40.594780 (+ 684us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:16:40.595043 (+ 263us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:16:40.595058 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:16:40.596785 (+ 1727us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:16:40.596792 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:16:40.598496 (+ 1704us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:16:40.598510 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:16:40.599727 (+ 1217us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:16:40.599734 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:16:40.599994 (+ 260us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:16:40.600776 (+ 782us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:16:40.600806 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:16:40.604565 (+ 3759us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:16:40.609949 (+ 5384us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:16:40.609956 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:16:40.609961 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:16:40.611017 (+ 1056us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:16:40.611668 (+ 651us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:16:40.611674 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:16:40.611679 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:16:40.612059 (+ 380us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:16:40.613064 (+ 1005us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:16:40.613069 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:16:40.613366 (+ 297us) client_negotiation.cc:769] Sending connection context
0928 02:16:40.613716 (+ 350us) client_negotiation.cc:240] Negotiation successful
0928 02:16:40.616737 (+ 3021us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":56,"mutex_wait_us":1}
I20260928 02:16:40.619259 29744 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 02:16:40.624866 29636 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P aab697abb3f64c20a42d6b6c684ea39f: Bootstrap starting.
I20260928 02:16:40.635973 29755 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20260928 02:16:40.638803 29717 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:16:40.580586 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39776 (local address 127.6.235.124:34699)
0928 02:16:40.580780 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0928 02:16:40.580787 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 02:16:40.584861 (+ 4074us) server_negotiation.cc:407] Connection header received
0928 02:16:40.585422 (+ 561us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:16:40.585428 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:16:40.585552 (+ 124us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:16:40.585636 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:16:40.588539 (+ 2903us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:16:40.589872 (+ 1333us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:16:40.602730 (+ 12858us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:16:40.603494 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:16:40.603604 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:16:40.629163 (+ 25559us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:16:40.629173 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:16:40.629181 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:16:40.629241 (+ 60us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:16:40.633249 (+ 4008us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:16:40.635270 (+ 2021us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:16:40.635278 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:16:40.635285 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:16:40.635476 (+ 191us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:16:40.636033 (+ 557us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:16:40.636039 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:16:40.636043 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:16:40.636657 (+ 614us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:16:40.636943 (+ 286us) server_negotiation.cc:1035] Waiting for connection context
0928 02:16:40.638119 (+ 1176us) server_negotiation.cc:299] Negotiation successful
0928 02:16:40.638388 (+ 269us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260928 02:16:40.639659 29636 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P aab697abb3f64c20a42d6b6c684ea39f: Neither blocks nor log segments found. Creating new log.
I20260928 02:16:40.641198 29636 log.cc:824] T 00000000000000000000000000000000 P aab697abb3f64c20a42d6b6c684ea39f: Log is configured to *not* fsync() on all Append() calls
W20260928 02:16:40.645205 29665 tablet.cc:2369] T 00000000000000000000000000000000 P bbc1fb6d83d04546aa8a14b43e8e8084: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 02:16:40.657258 29641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:16:40.562393 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59142 (local address 127.6.235.125:32979)
0928 02:16:40.562656 (+ 263us) server_negotiation.cc:206] Beginning negotiation
0928 02:16:40.562666 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 02:16:40.573636 (+ 10970us) server_negotiation.cc:407] Connection header received
0928 02:16:40.576108 (+ 2472us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:16:40.576119 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:16:40.576244 (+ 125us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:16:40.576347 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:16:40.588578 (+ 12231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:16:40.589857 (+ 1279us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:16:40.597172 (+ 7315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:16:40.597952 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:16:40.598058 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:16:40.625331 (+ 27273us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:16:40.625341 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:16:40.625346 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:16:40.625396 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:16:40.635057 (+ 9661us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:16:40.643407 (+ 8350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:16:40.643418 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:16:40.643423 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:16:40.653804 (+ 10381us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:16:40.655042 (+ 1238us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:16:40.655050 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:16:40.655053 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:16:40.655774 (+ 721us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:16:40.656068 (+ 294us) server_negotiation.cc:1035] Waiting for connection context
0928 02:16:40.656509 (+ 441us) server_negotiation.cc:299] Negotiation successful
0928 02:16:40.656776 (+ 267us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":90}
I20260928 02:16:40.661981 29636 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P aab697abb3f64c20a42d6b6c684ea39f: No bootstrap required, opened a new log
I20260928 02:16:40.663070 29715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:16:40.561690 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41350 (local address 127.6.235.126:33829)
0928 02:16:40.561936 (+ 246us) server_negotiation.cc:206] Beginning negotiation
0928 02:16:40.561943 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 02:16:40.580302 (+ 18359us) server_negotiation.cc:407] Connection header received
0928 02:16:40.583943 (+ 3641us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:16:40.583952 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:16:40.584071 (+ 119us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:16:40.584171 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:16:40.599091 (+ 14920us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:16:40.600445 (+ 1354us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:16:40.619516 (+ 19071us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:16:40.620324 (+ 808us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:16:40.620455 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:16:40.648839 (+ 28384us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:16:40.648850 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:16:40.648855 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:16:40.648928 (+ 73us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:16:40.658295 (+ 9367us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:16:40.659297 (+ 1002us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:16:40.659306 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:16:40.659313 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:16:40.659540 (+ 227us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:16:40.660021 (+ 481us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:16:40.660027 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:16:40.660030 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:16:40.660424 (+ 394us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:16:40.660761 (+ 337us) server_negotiation.cc:1035] Waiting for connection context
0928 02:16:40.662213 (+ 1452us) server_negotiation.cc:299] Negotiation successful
0928 02:16:40.662503 (+ 290us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260928 02:16:40.640672 29495 master_runner.cc:186] Error getting master registration for 127.6.235.125:32979: OK, instance_id { permanent_uuid: "aab697abb3f64c20a42d6b6c684ea39f" instance_seqno: 1790561799859704 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260928 02:16:40.669224 29495 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260928 02:16:40.673913 29636 raft_consensus.cc:348] T 00000000000000000000000000000000 P aab697abb3f64c20a42d6b6c684ea39f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f48dda22055547a596b2e2575946f3a4" member_type: VOTER last_known_addr { host: "127.6.235.126" port: 33829 } } peers { permanent_uuid: "aab697abb3f64c20a42d6b6c684ea39f" member_type: VOTER last_known_addr { host: "127.6.235.125" port: 32979 } } peers { permanent_uuid: "bbc1fb6d83d04546aa8a14b43e8e8084" member_type: VOTER last_known_addr { host: "127.6.235.124" port: 34699 } }
I20260928 02:16:40.674441 29636 raft_consensus.cc:729] T 00000000000000000000000000000000 P aab697abb3f64c20a42d6b6c684ea39f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aab697abb3f64c20a42d6b6c684ea39f, State: Initialized, Role: FOLLOWER
I20260928 02:16:40.675061 29636 consensus_queue.cc:260] T 00000000000000000000000000000000 P aab697abb3f64c20a42d6b6c684ea39f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f48dda22055547a596b2e2575946f3a4" member_type: VOTER last_known_addr { host: "127.6.235.126" port: 33829 } } peers { permanent_uuid: "aab697abb3f64c20a42d6b6c684ea39f" member_type: VOTER last_known_addr { host: "127.6.235.125" port: 32979 } } peers { permanent_uuid: "bbc1fb6d83d04546aa8a14b43e8e8084" member_type: VOTER last_known_addr { host: "127.6.235.124" port: 34699 } }
I20260928 02:16:40.682253 29758 sys_catalog.cc:455] T 00000000000000000000000000000000 P aab697abb3f64c20a42d6b6c684ea39f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f48dda22055547a596b2e2575946f3a4" member_type: VOTER last_known_addr { host: "127.6.235.126" port: 33829 } } peers { permanent_uuid: "aab697abb3f64c20a42d6b6c684ea39f" member_type: VOTER last_known_addr { host: "127.6.235.125" port: 32979 } } peers { permanent_uuid: "bbc1fb6d83d04546aa8a14b43e8e8084" member_type: VOTER last_known_addr { host: "127.6.235.124" port: 34699 } } }
I20260928 02:16:40.682757 29758 sys_catalog.cc:458] T 00000000000000000000000000000000 P aab697abb3f64c20a42d6b6c684ea39f [sys.catalog]: This master's current role is: FOLLOWER
I20260928 02:16:40.684908 29636 sys_catalog.cc:564] T 00000000000000000000000000000000 P aab697abb3f64c20a42d6b6c684ea39f [sys.catalog]: configured and running, proceeding with master startup.
I20260928 02:16:40.686111 29611 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "f48dda22055547a596b2e2575946f3a4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "aab697abb3f64c20a42d6b6c684ea39f"
I20260928 02:16:40.686676 29611 raft_consensus.cc:3037] T 00000000000000000000000000000000 P aab697abb3f64c20a42d6b6c684ea39f [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:16:40.688776 29686 raft_consensus.cc:1261] T 00000000000000000000000000000000 P bbc1fb6d83d04546aa8a14b43e8e8084 [term 1 FOLLOWER]: Refusing update from remote peer f48dda22055547a596b2e2575946f3a4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 02:16:40.690138 29735 consensus_queue.cc:1035] T 00000000000000000000000000000000 P f48dda22055547a596b2e2575946f3a4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bbc1fb6d83d04546aa8a14b43e8e8084" member_type: VOTER last_known_addr { host: "127.6.235.124" port: 34699 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 02:16:40.692775 29611 raft_consensus.cc:2454] T 00000000000000000000000000000000 P aab697abb3f64c20a42d6b6c684ea39f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f48dda22055547a596b2e2575946f3a4 in term 1.
I20260928 02:16:40.698120 29610 raft_consensus.cc:1261] T 00000000000000000000000000000000 P aab697abb3f64c20a42d6b6c684ea39f [term 1 FOLLOWER]: Refusing update from remote peer f48dda22055547a596b2e2575946f3a4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 02:16:40.700941 29735 consensus_queue.cc:1035] T 00000000000000000000000000000000 P f48dda22055547a596b2e2575946f3a4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "aab697abb3f64c20a42d6b6c684ea39f" member_type: VOTER last_known_addr { host: "127.6.235.125" port: 32979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 02:16:40.715922 29719 sys_catalog.cc:455] T 00000000000000000000000000000000 P bbc1fb6d83d04546aa8a14b43e8e8084 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f48dda22055547a596b2e2575946f3a4. Latest consensus state: current_term: 1 leader_uuid: "f48dda22055547a596b2e2575946f3a4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f48dda22055547a596b2e2575946f3a4" member_type: VOTER last_known_addr { host: "127.6.235.126" port: 33829 } } peers { permanent_uuid: "aab697abb3f64c20a42d6b6c684ea39f" member_type: VOTER last_known_addr { host: "127.6.235.125" port: 32979 } } peers { permanent_uuid: "bbc1fb6d83d04546aa8a14b43e8e8084" member_type: VOTER last_known_addr { host: "127.6.235.124" port: 34699 } } }
I20260928 02:16:40.716284 29719 sys_catalog.cc:458] T 00000000000000000000000000000000 P bbc1fb6d83d04546aa8a14b43e8e8084 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 02:16:40.720461 29761 mvcc.cc:205] Tried to move back new op lower bound from 7334141135608578048 to 7334141135057027072. Current Snapshot: MvccSnapshot[applied={T|T < 7334141135608578048}]
I20260928 02:16:40.720778 29757 mvcc.cc:205] Tried to move back new op lower bound from 7334141135608578048 to 7334141135057027072. Current Snapshot: MvccSnapshot[applied={T|T < 7334141135608578048}]
I20260928 02:16:40.723013 29719 sys_catalog.cc:455] T 00000000000000000000000000000000 P bbc1fb6d83d04546aa8a14b43e8e8084 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "f48dda22055547a596b2e2575946f3a4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f48dda22055547a596b2e2575946f3a4" member_type: VOTER last_known_addr { host: "127.6.235.126" port: 33829 } } peers { permanent_uuid: "aab697abb3f64c20a42d6b6c684ea39f" member_type: VOTER last_known_addr { host: "127.6.235.125" port: 32979 } } peers { permanent_uuid: "bbc1fb6d83d04546aa8a14b43e8e8084" member_type: VOTER last_known_addr { host: "127.6.235.124" port: 34699 } } }
I20260928 02:16:40.723342 29719 sys_catalog.cc:458] T 00000000000000000000000000000000 P bbc1fb6d83d04546aa8a14b43e8e8084 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 02:16:40.735709 29758 sys_catalog.cc:455] T 00000000000000000000000000000000 P aab697abb3f64c20a42d6b6c684ea39f [sys.catalog]: SysCatalogTable state changed. Reason: New leader f48dda22055547a596b2e2575946f3a4. Latest consensus state: current_term: 1 leader_uuid: "f48dda22055547a596b2e2575946f3a4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f48dda22055547a596b2e2575946f3a4" member_type: VOTER last_known_addr { host: "127.6.235.126" port: 33829 } } peers { permanent_uuid: "aab697abb3f64c20a42d6b6c684ea39f" member_type: VOTER last_known_addr { host: "127.6.235.125" port: 32979 } } peers { permanent_uuid: "bbc1fb6d83d04546aa8a14b43e8e8084" member_type: VOTER last_known_addr { host: "127.6.235.124" port: 34699 } } }
I20260928 02:16:40.736054 29758 sys_catalog.cc:458] T 00000000000000000000000000000000 P aab697abb3f64c20a42d6b6c684ea39f [sys.catalog]: This master's current role is: FOLLOWER
I20260928 02:16:40.741693 29733 sys_catalog.cc:455] T 00000000000000000000000000000000 P f48dda22055547a596b2e2575946f3a4 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "f48dda22055547a596b2e2575946f3a4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f48dda22055547a596b2e2575946f3a4" member_type: VOTER last_known_addr { host: "127.6.235.126" port: 33829 } } peers { permanent_uuid: "aab697abb3f64c20a42d6b6c684ea39f" member_type: VOTER last_known_addr { host: "127.6.235.125" port: 32979 } } peers { permanent_uuid: "bbc1fb6d83d04546aa8a14b43e8e8084" member_type: VOTER last_known_addr { host: "127.6.235.124" port: 34699 } } }
I20260928 02:16:40.742152 29733 sys_catalog.cc:458] T 00000000000000000000000000000000 P f48dda22055547a596b2e2575946f3a4 [sys.catalog]: This master's current role is: LEADER
I20260928 02:16:40.742602 29733 sys_catalog.cc:455] T 00000000000000000000000000000000 P f48dda22055547a596b2e2575946f3a4 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "f48dda22055547a596b2e2575946f3a4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f48dda22055547a596b2e2575946f3a4" member_type: VOTER last_known_addr { host: "127.6.235.126" port: 33829 } } peers { permanent_uuid: "aab697abb3f64c20a42d6b6c684ea39f" member_type: VOTER last_known_addr { host: "127.6.235.125" port: 32979 } } peers { permanent_uuid: "bbc1fb6d83d04546aa8a14b43e8e8084" member_type: VOTER last_known_addr { host: "127.6.235.124" port: 34699 } } }
I20260928 02:16:40.742924 29733 sys_catalog.cc:458] T 00000000000000000000000000000000 P f48dda22055547a596b2e2575946f3a4 [sys.catalog]: This master's current role is: LEADER
I20260928 02:16:40.746963 29744 catalog_manager.cc:1347] Generated new cluster ID: 4091baa54993428f9afe09e9c0b2f99b
I20260928 02:16:40.747288 29744 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 02:16:40.758134 29758 sys_catalog.cc:455] T 00000000000000000000000000000000 P aab697abb3f64c20a42d6b6c684ea39f [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "f48dda22055547a596b2e2575946f3a4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f48dda22055547a596b2e2575946f3a4" member_type: VOTER last_known_addr { host: "127.6.235.126" port: 33829 } } peers { permanent_uuid: "aab697abb3f64c20a42d6b6c684ea39f" member_type: VOTER last_known_addr { host: "127.6.235.125" port: 32979 } } peers { permanent_uuid: "bbc1fb6d83d04546aa8a14b43e8e8084" member_type: VOTER last_known_addr { host: "127.6.235.124" port: 34699 } } }
I20260928 02:16:40.758455 29758 sys_catalog.cc:458] T 00000000000000000000000000000000 P aab697abb3f64c20a42d6b6c684ea39f [sys.catalog]: This master's current role is: FOLLOWER
I20260928 02:16:40.765448 29776 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20260928 02:16:40.785817 29744 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 02:16:40.787807 29744 catalog_manager.cc:1504] Loading token signing keys...
I20260928 02:16:40.805287 29744 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f48dda22055547a596b2e2575946f3a4: Generated new TSK 0
I20260928 02:16:40.806488 29744 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260928 02:16:40.905400 29641 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:16:40.902591 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59146 (local address 127.6.235.125:32979)
0928 02:16:40.902854 (+ 263us) server_negotiation.cc:206] Beginning negotiation
0928 02:16:40.902865 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 02:16:40.903307 (+ 442us) server_negotiation.cc:407] Connection header received
0928 02:16:40.903620 (+ 313us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:16:40.903626 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:16:40.903723 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:16:40.903811 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:16:40.905155 (+ 1344us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59146: BlockingRecv error: recv got EOF from 127.0.0.1:59146 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
W20260928 02:16:40.907918 29715 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:16:40.903476 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41354 (local address 127.6.235.126:33829)
0928 02:16:40.903730 (+ 254us) server_negotiation.cc:206] Beginning negotiation
0928 02:16:40.903737 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 02:16:40.906190 (+ 2453us) server_negotiation.cc:407] Connection header received
0928 02:16:40.906438 (+ 248us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:16:40.906444 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:16:40.906556 (+ 112us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:16:40.906631 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:16:40.907659 (+ 1028us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41354: BlockingRecv error: recv got EOF from 127.0.0.1:41354 (error 108)
Metrics: {"server-negotiator.queue_time_us":92}
W20260928 02:16:40.911155 29717 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:16:40.909100 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39786 (local address 127.6.235.124:34699)
0928 02:16:40.909282 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0928 02:16:40.909288 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 02:16:40.909480 (+ 192us) server_negotiation.cc:407] Connection header received
0928 02:16:40.909655 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:16:40.909662 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:16:40.909739 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:16:40.909844 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:16:40.910857 (+ 1013us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39786: BlockingRecv error: recv got EOF from 127.0.0.1:39786 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
W20260928 02:16:40.919291 29641 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:16:40.916859 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59162 (local address 127.6.235.125:32979)
0928 02:16:40.917078 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0928 02:16:40.917085 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 02:16:40.917668 (+ 583us) server_negotiation.cc:407] Connection header received
0928 02:16:40.917929 (+ 261us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:16:40.917934 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:16:40.918076 (+ 142us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:16:40.918191 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:16:40.919019 (+ 828us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59162: BlockingRecv error: recv got EOF from 127.0.0.1:59162 (error 108)
Metrics: {"server-negotiator.queue_time_us":83}
W20260928 02:16:40.935271 29641 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:16:40.931091 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59164 (local address 127.6.235.125:32979)
0928 02:16:40.931300 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0928 02:16:40.931308 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 02:16:40.933651 (+ 2343us) server_negotiation.cc:407] Connection header received
0928 02:16:40.933886 (+ 235us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:16:40.933893 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:16:40.933992 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:16:40.934109 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:16:40.935043 (+ 934us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59164: BlockingRecv error: recv got EOF from 127.0.0.1:59164 (error 108)
Metrics: {"server-negotiator.queue_time_us":71}
W20260928 02:16:40.943044 29641 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:16:40.940596 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59170 (local address 127.6.235.125:32979)
0928 02:16:40.940844 (+ 248us) server_negotiation.cc:206] Beginning negotiation
0928 02:16:40.940851 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 02:16:40.941419 (+ 568us) server_negotiation.cc:407] Connection header received
0928 02:16:40.941634 (+ 215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:16:40.941639 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:16:40.941743 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:16:40.941846 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:16:40.942789 (+ 943us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59170: BlockingRecv error: recv got EOF from 127.0.0.1:59170 (error 108)
Metrics: {"server-negotiator.queue_time_us":113}
W20260928 02:16:40.957693 29641 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:16:40.955150 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59178 (local address 127.6.235.125:32979)
0928 02:16:40.955402 (+ 252us) server_negotiation.cc:206] Beginning negotiation
0928 02:16:40.955411 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 02:16:40.955993 (+ 582us) server_negotiation.cc:407] Connection header received
0928 02:16:40.956204 (+ 211us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:16:40.956210 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:16:40.956325 (+ 115us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:16:40.956442 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:16:40.957444 (+ 1002us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59178: BlockingRecv error: recv got EOF from 127.0.0.1:59178 (error 108)
Metrics: {"server-negotiator.queue_time_us":118}
W20260928 02:16:40.972304 29641 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:16:40.969875 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59190 (local address 127.6.235.125:32979)
0928 02:16:40.970079 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0928 02:16:40.970086 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 02:16:40.970698 (+ 612us) server_negotiation.cc:407] Connection header received
0928 02:16:40.970914 (+ 216us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:16:40.970922 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:16:40.971033 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:16:40.971149 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:16:40.972062 (+ 913us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59190: BlockingRecv error: recv got EOF from 127.0.0.1:59190 (error 108)
Metrics: {"server-negotiator.queue_time_us":72}
W20260928 02:16:40.995245 29641 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:16:40.992505 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59198 (local address 127.6.235.125:32979)
0928 02:16:40.992720 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0928 02:16:40.992727 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 02:16:40.993375 (+ 648us) server_negotiation.cc:407] Connection header received
0928 02:16:40.993645 (+ 270us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:16:40.993652 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:16:40.993750 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:16:40.993859 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:16:40.994990 (+ 1131us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59198: BlockingRecv error: recv got EOF from 127.0.0.1:59198 (error 108)
Metrics: {"server-negotiator.queue_time_us":81}
W20260928 02:16:41.037763 29641 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:16:41.035078 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59206 (local address 127.6.235.125:32979)
0928 02:16:41.035296 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0928 02:16:41.035303 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 02:16:41.035945 (+ 642us) server_negotiation.cc:407] Connection header received
0928 02:16:41.036204 (+ 259us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:16:41.036210 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:16:41.036329 (+ 119us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:16:41.036462 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:16:41.037475 (+ 1013us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59206: BlockingRecv error: recv got EOF from 127.0.0.1:59206 (error 108)
Metrics: {"server-negotiator.queue_time_us":75}
W20260928 02:16:41.112985 29641 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:16:41.110492 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59208 (local address 127.6.235.125:32979)
0928 02:16:41.110727 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0928 02:16:41.110735 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 02:16:41.111300 (+ 565us) server_negotiation.cc:407] Connection header received
0928 02:16:41.111519 (+ 219us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:16:41.111526 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:16:41.111627 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:16:41.111738 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:16:41.112723 (+ 985us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59208: BlockingRecv error: recv got EOF from 127.0.0.1:59208 (error 108)
Metrics: {"server-negotiator.queue_time_us":78}
W20260928 02:16:41.248620 29641 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:16:41.245418 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59222 (local address 127.6.235.125:32979)
0928 02:16:41.246526 (+ 1108us) server_negotiation.cc:206] Beginning negotiation
0928 02:16:41.246533 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 02:16:41.246591 (+ 58us) server_negotiation.cc:407] Connection header received
0928 02:16:41.246889 (+ 298us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:16:41.246896 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:16:41.247028 (+ 132us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:16:41.247162 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:16:41.248290 (+ 1128us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59222: BlockingRecv error: recv got EOF from 127.0.0.1:59222 (error 108)
Metrics: {"server-negotiator.queue_time_us":95}
W20260928 02:16:41.511818 29641 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:16:41.510066 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59198 (local address 127.6.235.125:32979)
0928 02:16:41.510280 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0928 02:16:41.510284 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:16:41.510518 (+ 234us) server_negotiation.cc:407] Connection header received
0928 02:16:41.510741 (+ 223us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:16:41.510745 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:16:41.510817 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:16:41.510918 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:16:41.511578 (+ 660us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59198: BlockingRecv error: recv got EOF from 127.0.0.1:59198 (error 108)
Metrics: {"server-negotiator.queue_time_us":80}
I20260928 02:16:41.526994 29731 catalog_manager.cc:1259] Loaded cluster ID: 4091baa54993428f9afe09e9c0b2f99b
I20260928 02:16:41.527144 29731 catalog_manager.cc:1552] T 00000000000000000000000000000000 P bbc1fb6d83d04546aa8a14b43e8e8084: loading cluster ID for follower catalog manager: success
I20260928 02:16:41.529944 29731 catalog_manager.cc:1574] T 00000000000000000000000000000000 P bbc1fb6d83d04546aa8a14b43e8e8084: acquiring CA information for follower catalog manager: success
I20260928 02:16:41.531713 29731 catalog_manager.cc:1602] T 00000000000000000000000000000000 P bbc1fb6d83d04546aa8a14b43e8e8084: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260928 02:16:41.770081 29776 catalog_manager.cc:1259] Loaded cluster ID: 4091baa54993428f9afe09e9c0b2f99b
I20260928 02:16:41.770293 29776 catalog_manager.cc:1552] T 00000000000000000000000000000000 P aab697abb3f64c20a42d6b6c684ea39f: loading cluster ID for follower catalog manager: success
I20260928 02:16:41.773607 29776 catalog_manager.cc:1574] T 00000000000000000000000000000000 P aab697abb3f64c20a42d6b6c684ea39f: acquiring CA information for follower catalog manager: success
I20260928 02:16:41.775664 29776 catalog_manager.cc:1602] T 00000000000000000000000000000000 P aab697abb3f64c20a42d6b6c684ea39f: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260928 02:16:42.036707 29843 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:16:42.029877 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59214 (local address 127.6.235.125:32979)
0928 02:16:42.030623 (+ 746us) server_negotiation.cc:206] Beginning negotiation
0928 02:16:42.030631 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 02:16:42.030651 (+ 20us) server_negotiation.cc:407] Connection header received
0928 02:16:42.030731 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:16:42.030741 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:16:42.030894 (+ 153us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:16:42.031052 (+ 158us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 02:16:42.031532 (+ 480us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:16:42.032445 (+ 913us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:16:42.033895 (+ 1450us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:16:42.034362 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:16:42.034504 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:16:42.035267 (+ 763us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 02:16:42.035961 (+ 694us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 02:16:42.036209 (+ 248us) server_negotiation.cc:1035] Waiting for connection context
0928 02:16:42.036269 (+ 60us) server_negotiation.cc:299] Negotiation successful
0928 02:16:42.036365 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":607,"thread_start_us":450,"threads_started":1}
I20260928 02:16:42.057282 29848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:16:42.043917 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53264 (local address 127.6.235.124:34699)
0928 02:16:42.051073 (+ 7156us) server_negotiation.cc:206] Beginning negotiation
0928 02:16:42.051081 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 02:16:42.051110 (+ 29us) server_negotiation.cc:407] Connection header received
0928 02:16:42.051264 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:16:42.051274 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:16:42.051546 (+ 272us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:16:42.051776 (+ 230us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 02:16:42.052492 (+ 716us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:16:42.053474 (+ 982us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:16:42.054878 (+ 1404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:16:42.055298 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:16:42.055437 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:16:42.055981 (+ 544us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 02:16:42.056568 (+ 587us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 02:16:42.056775 (+ 207us) server_negotiation.cc:1035] Waiting for connection context
0928 02:16:42.056838 (+ 63us) server_negotiation.cc:299] Negotiation successful
0928 02:16:42.056959 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6993,"thread_start_us":6880,"threads_started":1}
I20260928 02:16:42.087505 29855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:16:42.066948 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40280 (local address 127.6.235.126:33829)
0928 02:16:42.077396 (+ 10448us) server_negotiation.cc:206] Beginning negotiation
0928 02:16:42.077406 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 02:16:42.077435 (+ 29us) server_negotiation.cc:407] Connection header received
0928 02:16:42.077580 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:16:42.077587 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:16:42.077847 (+ 260us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:16:42.078117 (+ 270us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 02:16:42.079027 (+ 910us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:16:42.080133 (+ 1106us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:16:42.083317 (+ 3184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:16:42.084234 (+ 917us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:16:42.084388 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:16:42.085577 (+ 1189us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 02:16:42.086193 (+ 616us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 02:16:42.086993 (+ 800us) server_negotiation.cc:1035] Waiting for connection context
0928 02:16:42.087074 (+ 81us) server_negotiation.cc:299] Negotiation successful
0928 02:16:42.087168 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10290,"thread_start_us":10157,"threads_started":1}
I20260928 02:16:42.087891 29843 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:16:42.067119 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59220 (local address 127.6.235.125:32979)
0928 02:16:42.067353 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0928 02:16:42.067361 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 02:16:42.079594 (+ 12233us) server_negotiation.cc:407] Connection header received
0928 02:16:42.079917 (+ 323us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:16:42.079924 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:16:42.080026 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:16:42.080130 (+ 104us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 02:16:42.081287 (+ 1157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:16:42.082586 (+ 1299us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:16:42.085150 (+ 2564us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:16:42.086051 (+ 901us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:16:42.086168 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:16:42.086964 (+ 796us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 02:16:42.087127 (+ 163us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 02:16:42.087214 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0928 02:16:42.087460 (+ 246us) server_negotiation.cc:299] Negotiation successful
0928 02:16:42.087557 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":88}
I20260928 02:16:42.094406 7085 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskTlaUuc/build/asan/bin/kudu pid 29495
I20260928 02:16:42.094732 29848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:16:42.079845 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53276 (local address 127.6.235.124:34699)
0928 02:16:42.080036 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0928 02:16:42.080042 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 02:16:42.087118 (+ 7076us) server_negotiation.cc:407] Connection header received
0928 02:16:42.087701 (+ 583us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:16:42.087707 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:16:42.087806 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:16:42.087873 (+ 67us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 02:16:42.088632 (+ 759us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:16:42.089684 (+ 1052us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:16:42.092274 (+ 2590us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:16:42.093026 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:16:42.093189 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:16:42.093837 (+ 648us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 02:16:42.093989 (+ 152us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 02:16:42.094089 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0928 02:16:42.094359 (+ 270us) server_negotiation.cc:299] Negotiation successful
0928 02:16:42.094458 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260928 02:16:42.095614 29557 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 02:16:42.210345 7085 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTlaUuc/build/asan/bin/kudu with pid 29495
W20260928 02:16:42.234114 29585 connection.cc:535] server connection from 127.0.0.1:59110 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260928 02:16:42.234114 29660 connection.cc:535] server connection from 127.0.0.1:39762 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260928 02:16:42.234582 7085 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskTlaUuc/build/asan/bin/kudu pid 29569
I20260928 02:16:42.235785 29631 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 02:16:42.345781 7085 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTlaUuc/build/asan/bin/kudu with pid 29569
I20260928 02:16:42.367784 7085 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskTlaUuc/build/asan/bin/kudu pid 29644
I20260928 02:16:42.369078 29706 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 02:16:42.477463 7085 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTlaUuc/build/asan/bin/kudu with pid 29644
2026-09-28T02:16:42Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0 (5028 ms)
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1
Loading random data
Initializing database '/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790561432251112-7085-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 02:16:42 dist-test-slave-86gv krb5kdc[29864](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 02:16:42 dist-test-slave-86gv krb5kdc[29864](info): set up 2 sockets
Sep 28 02:16:42 dist-test-slave-86gv krb5kdc[29864](info): commencing operation
krb5kdc: starting...
W20260928 02:16:43.622123 7085 mini_kdc.cc:121] Time spent starting KDC: real 1.103s user 0.000s sys 0.033s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790561432251112-7085-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790561432251112-7085-0/krb5kdc/test-user.keytab.
Sep 28 02:16:43 dist-test-slave-86gv krb5kdc[29864](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561803, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T02:16:43Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T02:16:43Z Disabled control of system clock
WARNING: no policy specified for kudu/127.6.235.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.235.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.235.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790561432251112-7085-0/krb5kdc/kudu_127.6.235.126.keytab.
Entry for principal kudu/127.6.235.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790561432251112-7085-0/krb5kdc/kudu_127.6.235.126.keytab.
I20260928 02:16:43.917420 7085 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTlaUuc/build/asan/bin/kudu
/tmp/dist-test-taskTlaUuc/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790561432251112-7085-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790561432251112-7085-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790561432251112-7085-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790561432251112-7085-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.235.126:44965
--webserver_interface=127.6.235.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790561432251112-7085-0/krb5kdc/kudu_127.6.235.126.keytab
--principal=kudu/127.6.235.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.235.84:41181
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.235.126:44965,127.6.235.125:38915,127.6.235.124:34049
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790561432251112-7085-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790561432251112-7085-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790561432251112-7085-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:16:44.282946 29877 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260928 02:16:44.283387 29877 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:16:44.283537 29877 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:16:44.283691 29877 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:16:44.292116 29877 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 02:16:44.292268 29877 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:16:44.292402 29877 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:16:44.292500 29877 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 02:16:44.292627 29877 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 02:16:44.304235 29877 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.235.84:41181
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790561432251112-7085-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790561432251112-7085-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.235.126:44965,127.6.235.125:38915,127.6.235.124:34049
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790561432251112-7085-0/krb5kdc/kudu_127.6.235.126.keytab
--principal=kudu/127.6.235.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.235.126:44965
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790561432251112-7085-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.235.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790561432251112-7085-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 00:51:59 UTC on ccycloud.quasar-axgkwn.root.comops.site
build id 4676464
ASAN enabled
I20260928 02:16:44.305893 29877 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:16:44.307381 29877 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:16:44.321254 29885 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:16:44.321230 29883 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:16:44.321309 29882 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:16:44.322232 29877 server_base.cc:1034] running on GCE node
I20260928 02:16:44.323104 29877 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:16:44.324591 29877 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:16:44.325853 29877 hybrid_clock.cc:648] HybridClock initialized: now 1790561804325767 us; error 55 us; skew 500 ppm
I20260928 02:16:44.326260 29877 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 02:16:44 dist-test-slave-86gv krb5kdc[29864](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561804, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.235.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:16:44.329730 29877 init.cc:377] Logged in from keytab as kudu/127.6.235.126@KRBTEST.COM (short username kudu)
I20260928 02:16:44.331275 29877 webserver.cc:460] Webserver started at http://127.6.235.126:44123/ using document root <none> and password file <none>
I20260928 02:16:44.331970 29877 fs_manager.cc:359] Metadata directory not provided
I20260928 02:16:44.332062 29877 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:16:44.332340 29877 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:16:44.335541 29877 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790561432251112-7085-0/minicluster-data/master-0/data/instance:
uuid: "69d84461ef5b4b29aaf2cd81fd158769"
format_stamp: "Formatted at 2026-09-28 02:16:44 on dist-test-slave-86gv"
server_key: "9d6ef9ae6726b7b4c1cbca57ab86e7a2"
server_key_iv: "8fc3c2d8ff2b32d23ba65cdd57609871"
server_key_version: "encryptionkey@0"
I20260928 02:16:44.336377 29877 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790561432251112-7085-0/minicluster-data/master-0/wal/instance:
uuid: "69d84461ef5b4b29aaf2cd81fd158769"
format_stamp: "Formatted at 2026-09-28 02:16:44 on dist-test-slave-86gv"
server_key: "9d6ef9ae6726b7b4c1cbca57ab86e7a2"
server_key_iv: "8fc3c2d8ff2b32d23ba65cdd57609871"
server_key_version: "encryptionkey@0"
I20260928 02:16:44.342444 29877 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.002s
I20260928 02:16:44.346702 29892 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:16:44.348429 29877 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.000s sys 0.004s
I20260928 02:16:44.348664 29877 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790561432251112-7085-0/minicluster-data/master-0/data,/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790561432251112-7085-0/minicluster-data/master-0/wal
uuid: "69d84461ef5b4b29aaf2cd81fd158769"
format_stamp: "Formatted at 2026-09-28 02:16:44 on dist-test-slave-86gv"
server_key: "9d6ef9ae6726b7b4c1cbca57ab86e7a2"
server_key_iv: "8fc3c2d8ff2b32d23ba65cdd57609871"
server_key_version: "encryptionkey@0"
I20260928 02:16:44.348965 29877 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790561432251112-7085-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790561432251112-7085-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790561432251112-7085-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:16:44.379113 29877 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:16:44.384063 29877 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:16:44.384392 29877 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:16:44.404009 29877 rpc_server.cc:307] RPC server started. Bound to: 127.6.235.126:44965
I20260928 02:16:44.404029 29943 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.235.126:44965 every 8 connection(s)
I20260928 02:16:44.405546 29877 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790561432251112-7085-0/minicluster-data/master-0/data/info.pb
I20260928 02:16:44.409308 7085 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTlaUuc/build/asan/bin/kudu as pid 29877
I20260928 02:16:44.409507 7085 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790561432251112-7085-0/minicluster-data/master-0/wal/instance
I20260928 02:16:44.409755 29944 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:16:44.410010 7085 external_mini_cluster.cc:1428] Setting key b744d3844d0c9d9eebe1e07d81accd88
I20260928 02:16:44.416815 29944 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.6.235.126" port: 44965 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 02:16:44 dist-test-slave-86gv krb5kdc[29864](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561804, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.235.126@KRBTEST.COM for kudu/127.6.235.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.6.235.125@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.235.125@KRBTEST.COM" created.
I20260928 02:16:44.444398 29948 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:16:44.420559 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58948 (local address 127.6.235.126:44965)
0928 02:16:44.425493 (+ 4934us) server_negotiation.cc:206] Beginning negotiation
0928 02:16:44.425504 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 02:16:44.425555 (+ 51us) server_negotiation.cc:407] Connection header received
0928 02:16:44.425740 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:16:44.425750 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:16:44.426100 (+ 350us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:16:44.426312 (+ 212us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:16:44.427576 (+ 1264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:16:44.429539 (+ 1963us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:16:44.431031 (+ 1492us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:16:44.432068 (+ 1037us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:16:44.432279 (+ 211us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:16:44.436390 (+ 4111us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:16:44.436399 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:16:44.436413 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:16:44.436445 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:16:44.439898 (+ 3453us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:16:44.440645 (+ 747us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:16:44.440657 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:16:44.440663 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:16:44.440860 (+ 197us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:16:44.441312 (+ 452us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:16:44.441320 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:16:44.441326 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:16:44.442105 (+ 779us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:16:44.442310 (+ 205us) server_negotiation.cc:1035] Waiting for connection context
0928 02:16:44.442933 (+ 623us) server_negotiation.cc:299] Negotiation successful
0928 02:16:44.443385 (+ 452us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":161,"server-negotiator.queue_time_us":4693,"spinlock_wait_cycles":42880,"thread_start_us":4452,"threads_started":1}
I20260928 02:16:44.444453 29947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:16:44.420559 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.235.126:44965 (local address 127.0.0.1:58948)
0928 02:16:44.421565 (+ 1006us) negotiation.cc:107] Waiting for socket to connect
0928 02:16:44.421622 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0928 02:16:44.424563 (+ 2941us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:16:44.426368 (+ 1805us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:16:44.426374 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:16:44.426968 (+ 594us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:16:44.427328 (+ 360us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:16:44.427347 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:16:44.429706 (+ 2359us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:16:44.429710 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:16:44.430863 (+ 1153us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:16:44.430876 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:16:44.432296 (+ 1420us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:16:44.432301 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:16:44.432513 (+ 212us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:16:44.433300 (+ 787us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:16:44.433323 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:16:44.436114 (+ 2791us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:16:44.440046 (+ 3932us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:16:44.440062 (+ 16us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:16:44.440078 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:16:44.440480 (+ 402us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:16:44.441040 (+ 560us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:16:44.441045 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:16:44.441047 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:16:44.441198 (+ 151us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:16:44.442300 (+ 1102us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:16:44.442307 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:16:44.442513 (+ 206us) client_negotiation.cc:769] Sending connection context
0928 02:16:44.442750 (+ 237us) client_negotiation.cc:240] Negotiation successful
0928 02:16:44.443216 (+ 466us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":664,"mutex_wait_us":61,"thread_start_us":436,"threads_started":1}
I20260928 02:16:44.447567 29944 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.6.235.125" port: 38915 } has no permanent_uuid. Determining permanent_uuid...
I20260928 02:16:44.448743 29947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:16:44.448300 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.235.125:38915 (local address 127.0.0.1:56854)
0928 02:16:44.448488 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0928 02:16:44.448614 (+ 126us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.235.125:38915: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
W20260928 02:16:44.449126 29896 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.235.125:38915: connect: Connection refused (error 111)
I20260928 02:16:44.450939 29947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:16:44.450395 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.235.125:38915 (local address 127.0.0.1:56866)
0928 02:16:44.450618 (+ 223us) negotiation.cc:107] Waiting for socket to connect
0928 02:16:44.450754 (+ 136us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.235.125:38915: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
W20260928 02:16:44.451160 29944 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.6.235.125:38915: Network error: Client connection negotiation failed: client connection to 127.6.235.125:38915: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.235.125 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790561432251112-7085-0/krb5kdc/kudu_127.6.235.125.keytab.
Entry for principal kudu/127.6.235.125 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790561432251112-7085-0/krb5kdc/kudu_127.6.235.125.keytab.
I20260928 02:16:44.477787 7085 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTlaUuc/build/asan/bin/kudu
/tmp/dist-test-taskTlaUuc/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790561432251112-7085-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790561432251112-7085-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790561432251112-7085-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790561432251112-7085-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.235.125:38915
--webserver_interface=127.6.235.125
--webserver_port=0
--keytab_file=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790561432251112-7085-0/krb5kdc/kudu_127.6.235.125.keytab
--principal=kudu/127.6.235.125
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.235.84:41181
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.235.126:44965,127.6.235.125:38915,127.6.235.124:34049
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790561432251112-7085-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790561432251112-7085-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790561432251112-7085-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 02:16:44.500548 29944 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.235.125" port: 38915 } attempt: 1
I20260928 02:16:44.501891 29947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:16:44.501409 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.235.125:38915 (local address 127.0.0.1:56876)
0928 02:16:44.501631 (+ 222us) negotiation.cc:107] Waiting for socket to connect
0928 02:16:44.501745 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.235.125:38915: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":85}
I20260928 02:16:44.503145 29947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:16:44.502721 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.235.125:38915 (local address 127.0.0.1:56880)
0928 02:16:44.502918 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0928 02:16:44.502998 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.235.125:38915: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":91}
W20260928 02:16:44.503387 29944 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.6.235.125:38915: Network error: Client connection negotiation failed: client connection to 127.6.235.125:38915: connect: Connection refused (error 111)
I20260928 02:16:44.571692 29944 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.235.125" port: 38915 } attempt: 2
I20260928 02:16:44.573246 29947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:16:44.572521 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.235.125:38915 (local address 127.0.0.1:56884)
0928 02:16:44.572842 (+ 321us) negotiation.cc:107] Waiting for socket to connect
0928 02:16:44.573012 (+ 170us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.235.125:38915: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":126}
I20260928 02:16:44.574368 29947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:16:44.573978 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.235.125:38915 (local address 127.0.0.1:56890)
0928 02:16:44.574138 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0928 02:16:44.574242 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.235.125:38915: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
W20260928 02:16:44.574556 29944 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.6.235.125:38915: Network error: Client connection negotiation failed: client connection to 127.6.235.125:38915: connect: Connection refused (error 111)
I20260928 02:16:44.665885 29944 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.235.125" port: 38915 } attempt: 3
I20260928 02:16:44.667109 29947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:16:44.666649 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.235.125:38915 (local address 127.0.0.1:56896)
0928 02:16:44.666859 (+ 210us) negotiation.cc:107] Waiting for socket to connect
0928 02:16:44.666971 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.235.125:38915: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":81}
I20260928 02:16:44.668283 29947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:16:44.667888 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.235.125:38915 (local address 127.0.0.1:56902)
0928 02:16:44.668066 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0928 02:16:44.668142 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.235.125:38915: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":73}
W20260928 02:16:44.668469 29944 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.6.235.125:38915: Network error: Client connection negotiation failed: client connection to 127.6.235.125:38915: connect: Connection refused (error 111)
I20260928 02:16:44.811795 29944 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.235.125" port: 38915 } attempt: 4
I20260928 02:16:44.813122 29947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:16:44.812509 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.235.125:38915 (local address 127.0.0.1:56908)
0928 02:16:44.812831 (+ 322us) negotiation.cc:107] Waiting for socket to connect
0928 02:16:44.812976 (+ 145us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.235.125:38915: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":160}
I20260928 02:16:44.814224 29947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:16:44.813858 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.235.125:38915 (local address 127.0.0.1:56918)
0928 02:16:44.814018 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0928 02:16:44.814096 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.235.125:38915: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
W20260928 02:16:44.814484 29944 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.6.235.125:38915: Network error: Client connection negotiation failed: client connection to 127.6.235.125:38915: connect: Connection refused (error 111)
W20260928 02:16:44.832477 29951 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260928 02:16:44.832891 29951 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:16:44.833066 29951 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:16:44.833207 29951 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:16:44.840709 29951 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 02:16:44.840814 29951 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:16:44.840911 29951 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:16:44.841086 29951 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 02:16:44.841205 29951 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 02:16:44.852288 29951 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.235.84:41181
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790561432251112-7085-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790561432251112-7085-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.6.235.126:44965,127.6.235.125:38915,127.6.235.124:34049
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790561432251112-7085-0/krb5kdc/kudu_127.6.235.125.keytab
--principal=kudu/127.6.235.125
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.235.125:38915
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790561432251112-7085-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.6.235.125
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790561432251112-7085-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 00:51:59 UTC on ccycloud.quasar-axgkwn.root.comops.site
build id 4676464
ASAN enabled
I20260928 02:16:44.853899 29951 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:16:44.855365 29951 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:16:44.868808 29959 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:16:44.868705 29957 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:16:44.868966 29956 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:16:44.869900 29951 server_base.cc:1034] running on GCE node
I20260928 02:16:44.870849 29951 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:16:44.872506 29951 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:16:44.873807 29951 hybrid_clock.cc:648] HybridClock initialized: now 1790561804873734 us; error 49 us; skew 500 ppm
I20260928 02:16:44.874257 29951 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 02:16:44 dist-test-slave-86gv krb5kdc[29864](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561804, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.235.125@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:16:44.878007 29951 init.cc:377] Logged in from keytab as kudu/127.6.235.125@KRBTEST.COM (short username kudu)
I20260928 02:16:44.879842 29951 webserver.cc:460] Webserver started at http://127.6.235.125:36869/ using document root <none> and password file <none>
I20260928 02:16:44.880602 29951 fs_manager.cc:359] Metadata directory not provided
I20260928 02:16:44.880699 29951 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:16:44.881065 29951 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:16:44.884315 29951 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790561432251112-7085-0/minicluster-data/master-1/data/instance:
uuid: "387b63f25d984ebfa73f753ff65b2d8f"
format_stamp: "Formatted at 2026-09-28 02:16:44 on dist-test-slave-86gv"
server_key: "c927250bfa89d0d9097e1a2f0b239ddd"
server_key_iv: "0d4c5e544116c0241d7dced9e0c30d83"
server_key_version: "encryptionkey@0"
I20260928 02:16:44.885300 29951 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790561432251112-7085-0/minicluster-data/master-1/wal/instance:
uuid: "387b63f25d984ebfa73f753ff65b2d8f"
format_stamp: "Formatted at 2026-09-28 02:16:44 on dist-test-slave-86gv"
server_key: "c927250bfa89d0d9097e1a2f0b239ddd"
server_key_iv: "0d4c5e544116c0241d7dced9e0c30d83"
server_key_version: "encryptionkey@0"
I20260928 02:16:44.891583 29951 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260928 02:16:44.895388 29966 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:16:44.897112 29951 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 02:16:44.897313 29951 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790561432251112-7085-0/minicluster-data/master-1/data,/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790561432251112-7085-0/minicluster-data/master-1/wal
uuid: "387b63f25d984ebfa73f753ff65b2d8f"
format_stamp: "Formatted at 2026-09-28 02:16:44 on dist-test-slave-86gv"
server_key: "c927250bfa89d0d9097e1a2f0b239ddd"
server_key_iv: "0d4c5e544116c0241d7dced9e0c30d83"
server_key_version: "encryptionkey@0"
I20260928 02:16:44.897639 29951 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790561432251112-7085-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790561432251112-7085-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790561432251112-7085-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:16:44.942777 29951 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:16:44.947541 29951 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:16:44.947854 29951 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:16:44.967634 29951 rpc_server.cc:307] RPC server started. Bound to: 127.6.235.125:38915
I20260928 02:16:44.967656 30017 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.235.125:38915 every 8 connection(s)
I20260928 02:16:44.969187 29951 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790561432251112-7085-0/minicluster-data/master-1/data/info.pb
I20260928 02:16:44.970260 7085 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTlaUuc/build/asan/bin/kudu as pid 29951
I20260928 02:16:44.970456 7085 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790561432251112-7085-0/minicluster-data/master-1/wal/instance
I20260928 02:16:44.970999 7085 external_mini_cluster.cc:1428] Setting key e30d0f21d0a3faf3235430052109b7f7
I20260928 02:16:44.973433 30018 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:16:44.980350 30018 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.6.235.126" port: 44965 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.6.235.124@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.235.124@KRBTEST.COM" created.
Sep 28 02:16:44 dist-test-slave-86gv krb5kdc[29864](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561804, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.235.125@KRBTEST.COM for kudu/127.6.235.126@KRBTEST.COM
I20260928 02:16:45.010070 30021 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:16:44.984185 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58958 (local address 127.6.235.126:44965)
0928 02:16:44.984822 (+ 637us) server_negotiation.cc:206] Beginning negotiation
0928 02:16:44.984829 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 02:16:44.985924 (+ 1095us) server_negotiation.cc:407] Connection header received
0928 02:16:44.989050 (+ 3126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:16:44.989055 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:16:44.989147 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:16:44.989253 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:16:44.991690 (+ 2437us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:16:44.992551 (+ 861us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:16:44.995292 (+ 2741us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:16:44.995731 (+ 439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:16:44.995842 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:16:45.003616 (+ 7774us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:16:45.003622 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:16:45.003626 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:16:45.003674 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:16:45.006560 (+ 2886us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:16:45.007450 (+ 890us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:16:45.007455 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:16:45.007458 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:16:45.007585 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:16:45.008049 (+ 464us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:16:45.008053 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:16:45.008061 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:16:45.008370 (+ 309us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:16:45.008551 (+ 181us) server_negotiation.cc:1035] Waiting for connection context
0928 02:16:45.009416 (+ 865us) server_negotiation.cc:299] Negotiation successful
0928 02:16:45.009672 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":494,"thread_start_us":290,"threads_started":1}
I20260928 02:16:45.011576 30022 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:16:44.984765 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.235.126:44965 (local address 127.0.0.1:58958)
0928 02:16:44.985720 (+ 955us) negotiation.cc:107] Waiting for socket to connect
0928 02:16:44.985778 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0928 02:16:44.988623 (+ 2845us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:16:44.989605 (+ 982us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:16:44.989619 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:16:44.990510 (+ 891us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:16:44.991400 (+ 890us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:16:44.991431 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:16:44.992757 (+ 1326us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:16:44.992766 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:16:44.995093 (+ 2327us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:16:44.995115 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:16:44.996009 (+ 894us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:16:44.996019 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:16:44.996419 (+ 400us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:16:44.997301 (+ 882us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:16:44.997339 (+ 38us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:16:45.003313 (+ 5974us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:16:45.006757 (+ 3444us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:16:45.006768 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:16:45.006789 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:16:45.007289 (+ 500us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:16:45.007707 (+ 418us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:16:45.007712 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:16:45.007716 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:16:45.007918 (+ 202us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:16:45.008536 (+ 618us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:16:45.008545 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:16:45.009007 (+ 462us) client_negotiation.cc:769] Sending connection context
0928 02:16:45.009322 (+ 315us) client_negotiation.cc:240] Negotiation successful
0928 02:16:45.009895 (+ 573us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":642,"thread_start_us":323,"threads_started":1}
I20260928 02:16:45.013969 30018 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.6.235.125" port: 38915 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 02:16:45 dist-test-slave-86gv krb5kdc[29864](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561804, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.235.125@KRBTEST.COM for kudu/127.6.235.125@KRBTEST.COM
I20260928 02:16:45.031620 30022 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:16:45.014843 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.235.125:38915 (local address 127.0.0.1:56924)
0928 02:16:45.015105 (+ 262us) negotiation.cc:107] Waiting for socket to connect
0928 02:16:45.015129 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0928 02:16:45.015267 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:16:45.016340 (+ 1073us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:16:45.016349 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:16:45.016996 (+ 647us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:16:45.017238 (+ 242us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:16:45.017254 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:16:45.018815 (+ 1561us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:16:45.018823 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:16:45.020486 (+ 1663us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:16:45.020505 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:16:45.021522 (+ 1017us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:16:45.021532 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:16:45.021794 (+ 262us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:16:45.022531 (+ 737us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:16:45.022563 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:16:45.025436 (+ 2873us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:16:45.028872 (+ 3436us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:16:45.028880 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:16:45.028884 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:16:45.029292 (+ 408us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:16:45.029736 (+ 444us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:16:45.029741 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:16:45.029744 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:16:45.029869 (+ 125us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:16:45.030506 (+ 637us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:16:45.030510 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:16:45.030707 (+ 197us) client_negotiation.cc:769] Sending connection context
0928 02:16:45.030846 (+ 139us) client_negotiation.cc:240] Negotiation successful
0928 02:16:45.031149 (+ 303us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":77}
I20260928 02:16:45.031661 30024 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:16:45.015118 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56924 (local address 127.6.235.125:38915)
0928 02:16:45.015872 (+ 754us) server_negotiation.cc:206] Beginning negotiation
0928 02:16:45.015878 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 02:16:45.015904 (+ 26us) server_negotiation.cc:407] Connection header received
0928 02:16:45.015949 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:16:45.015955 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:16:45.016149 (+ 194us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:16:45.016271 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:16:45.017458 (+ 1187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:16:45.018661 (+ 1203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:16:45.020667 (+ 2006us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:16:45.021343 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:16:45.021445 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:16:45.025676 (+ 4231us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:16:45.025689 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:16:45.025705 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:16:45.025732 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:16:45.028705 (+ 2973us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:16:45.029435 (+ 730us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:16:45.029444 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:16:45.029447 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:16:45.029577 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:16:45.029985 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:16:45.029990 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:16:45.029993 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:16:45.030367 (+ 374us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:16:45.030616 (+ 249us) server_negotiation.cc:1035] Waiting for connection context
0928 02:16:45.030988 (+ 372us) server_negotiation.cc:299] Negotiation successful
0928 02:16:45.031270 (+ 282us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":38,"server-negotiator.queue_time_us":619,"thread_start_us":405,"threads_started":1}
I20260928 02:16:45.034387 30018 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.6.235.124" port: 34049 } has no permanent_uuid. Determining permanent_uuid...
I20260928 02:16:45.035506 30022 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:16:45.035057 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.235.124:34049 (local address 127.0.0.1:38200)
0928 02:16:45.035227 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0928 02:16:45.035336 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.235.124:34049: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
W20260928 02:16:45.035794 29967 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.235.124:34049: connect: Connection refused (error 111)
I20260928 02:16:45.037511 30022 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:16:45.037165 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.235.124:34049 (local address 127.0.0.1:38212)
0928 02:16:45.037331 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0928 02:16:45.037401 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.235.124:34049: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260928 02:16:45.037774 30018 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.6.235.124:34049: Network error: Client connection negotiation failed: client connection to 127.6.235.124:34049: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.235.124 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790561432251112-7085-0/krb5kdc/kudu_127.6.235.124.keytab.
Entry for principal kudu/127.6.235.124 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790561432251112-7085-0/krb5kdc/kudu_127.6.235.124.keytab.
I20260928 02:16:45.041718 7085 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskTlaUuc/build/asan/bin/kudu
/tmp/dist-test-taskTlaUuc/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790561432251112-7085-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790561432251112-7085-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790561432251112-7085-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790561432251112-7085-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.235.124:34049
--webserver_interface=127.6.235.124
--webserver_port=0
--keytab_file=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790561432251112-7085-0/krb5kdc/kudu_127.6.235.124.keytab
--principal=kudu/127.6.235.124
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.235.84:41181
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.235.126:44965,127.6.235.125:38915,127.6.235.124:34049
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790561432251112-7085-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790561432251112-7085-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790561432251112-7085-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 02:16:45.087224 30018 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.235.124" port: 34049 } attempt: 1
I20260928 02:16:45.088680 30022 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:16:45.088021 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.235.124:34049 (local address 127.0.0.1:38214)
0928 02:16:45.088325 (+ 304us) negotiation.cc:107] Waiting for socket to connect
0928 02:16:45.088456 (+ 131us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.235.124:34049: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":118}
I20260928 02:16:45.089882 30022 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:16:45.089482 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.235.124:34049 (local address 127.0.0.1:38218)
0928 02:16:45.089689 (+ 207us) negotiation.cc:107] Waiting for socket to connect
0928 02:16:45.089779 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.235.124:34049: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":112}
W20260928 02:16:45.090210 30018 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.6.235.124:34049: Network error: Client connection negotiation failed: client connection to 127.6.235.124:34049: connect: Connection refused (error 111)
I20260928 02:16:45.113756 29944 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.235.125" port: 38915 } attempt: 5
Sep 28 02:16:45 dist-test-slave-86gv krb5kdc[29864](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561804, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.235.126@KRBTEST.COM for kudu/127.6.235.125@KRBTEST.COM
I20260928 02:16:45.129196 29947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:16:45.114501 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.235.125:38915 (local address 127.0.0.1:56932)
0928 02:16:45.114803 (+ 302us) negotiation.cc:107] Waiting for socket to connect
0928 02:16:45.114831 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0928 02:16:45.114986 (+ 155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:16:45.115528 (+ 542us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:16:45.115533 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:16:45.116032 (+ 499us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:16:45.116231 (+ 199us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:16:45.116243 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:16:45.117520 (+ 1277us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:16:45.117528 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:16:45.118431 (+ 903us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:16:45.118447 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:16:45.119289 (+ 842us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:16:45.119295 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:16:45.119544 (+ 249us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:16:45.120242 (+ 698us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:16:45.120260 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:16:45.122475 (+ 2215us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:16:45.126139 (+ 3664us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:16:45.126154 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:16:45.126159 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:16:45.126717 (+ 558us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:16:45.127178 (+ 461us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:16:45.127184 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:16:45.127188 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:16:45.127343 (+ 155us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:16:45.128018 (+ 675us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:16:45.128023 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:16:45.128200 (+ 177us) client_negotiation.cc:769] Sending connection context
0928 02:16:45.128377 (+ 177us) client_negotiation.cc:240] Negotiation successful
0928 02:16:45.128734 (+ 357us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":79}
I20260928 02:16:45.129196 30024 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:16:45.114731 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56932 (local address 127.6.235.125:38915)
0928 02:16:45.114909 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0928 02:16:45.114914 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 02:16:45.114933 (+ 19us) server_negotiation.cc:407] Connection header received
0928 02:16:45.115208 (+ 275us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:16:45.115214 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:16:45.115314 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:16:45.115487 (+ 173us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:16:45.116417 (+ 930us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:16:45.117357 (+ 940us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:16:45.118604 (+ 1247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:16:45.119116 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:16:45.119276 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:16:45.122693 (+ 3417us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:16:45.122699 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:16:45.122702 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:16:45.122735 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:16:45.125891 (+ 3156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:16:45.126872 (+ 981us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:16:45.126880 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:16:45.126883 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:16:45.127045 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:16:45.127492 (+ 447us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:16:45.127501 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:16:45.127506 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:16:45.127840 (+ 334us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:16:45.128059 (+ 219us) server_negotiation.cc:1035] Waiting for connection context
0928 02:16:45.128498 (+ 439us) server_negotiation.cc:299] Negotiation successful
0928 02:16:45.128739 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260928 02:16:45.130403 29944 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.6.235.124" port: 34049 } has no permanent_uuid. Determining permanent_uuid...
I20260928 02:16:45.131615 29947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:16:45.131093 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.235.124:34049 (local address 127.0.0.1:38230)
0928 02:16:45.131267 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0928 02:16:45.131353 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.235.124:34049: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":69}
I20260928 02:16:45.132696 29947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:16:45.132331 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.235.124:34049 (local address 127.0.0.1:38234)
0928 02:16:45.132504 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0928 02:16:45.132588 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.235.124:34049: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":74}
W20260928 02:16:45.132995 29944 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.6.235.124:34049: Network error: Client connection negotiation failed: client connection to 127.6.235.124:34049: connect: Connection refused (error 111)
I20260928 02:16:45.158463 30018 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.235.124" port: 34049 } attempt: 2
I20260928 02:16:45.159833 30022 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:16:45.159279 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.235.124:34049 (local address 127.0.0.1:38250)
0928 02:16:45.159572 (+ 293us) negotiation.cc:107] Waiting for socket to connect
0928 02:16:45.159674 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.235.124:34049: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":95}
I20260928 02:16:45.161105 30022 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:16:45.160608 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.235.124:34049 (local address 127.0.0.1:38254)
0928 02:16:45.160819 (+ 211us) negotiation.cc:107] Waiting for socket to connect
0928 02:16:45.160947 (+ 128us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.235.124:34049: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
W20260928 02:16:45.161321 30018 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.6.235.124:34049: Network error: Client connection negotiation failed: client connection to 127.6.235.124:34049: connect: Connection refused (error 111)
I20260928 02:16:45.184247 29944 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.235.124" port: 34049 } attempt: 1
I20260928 02:16:45.185580 29947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:16:45.185014 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.235.124:34049 (local address 127.0.0.1:38260)
0928 02:16:45.185245 (+ 231us) negotiation.cc:107] Waiting for socket to connect
0928 02:16:45.185363 (+ 118us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.235.124:34049: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":75}
I20260928 02:16:45.186772 29947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:16:45.186358 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.235.124:34049 (local address 127.0.0.1:38276)
0928 02:16:45.186535 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0928 02:16:45.186624 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.235.124:34049: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
W20260928 02:16:45.187037 29944 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.6.235.124:34049: Network error: Client connection negotiation failed: client connection to 127.6.235.124:34049: connect: Connection refused (error 111)
I20260928 02:16:45.252652 30018 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.235.124" port: 34049 } attempt: 3
I20260928 02:16:45.254037 30022 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:16:45.253440 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.235.124:34049 (local address 127.0.0.1:38286)
0928 02:16:45.253769 (+ 329us) negotiation.cc:107] Waiting for socket to connect
0928 02:16:45.253878 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.235.124:34049: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":147}
I20260928 02:16:45.255292 29944 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.235.124" port: 34049 } attempt: 2
I20260928 02:16:45.255553 30022 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:16:45.254874 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.235.124:34049 (local address 127.0.0.1:38300)
0928 02:16:45.255303 (+ 429us) negotiation.cc:107] Waiting for socket to connect
0928 02:16:45.255407 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.235.124:34049: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":279}
W20260928 02:16:45.255779 30018 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.6.235.124:34049: Network error: Client connection negotiation failed: client connection to 127.6.235.124:34049: connect: Connection refused (error 111)
I20260928 02:16:45.256381 29947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:16:45.255963 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.235.124:34049 (local address 127.0.0.1:38302)
0928 02:16:45.256154 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0928 02:16:45.256241 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.235.124:34049: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":71}
I20260928 02:16:45.257515 29947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:16:45.257149 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.235.124:34049 (local address 127.0.0.1:38310)
0928 02:16:45.257332 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0928 02:16:45.257402 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.235.124:34049: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":86}
W20260928 02:16:45.257844 29944 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.6.235.124:34049: Network error: Client connection negotiation failed: client connection to 127.6.235.124:34049: connect: Connection refused (error 111)
I20260928 02:16:45.364161 29944 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.235.124" port: 34049 } attempt: 3
I20260928 02:16:45.365684 29947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:16:45.365133 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.235.124:34049 (local address 127.0.0.1:38322)
0928 02:16:45.365363 (+ 230us) negotiation.cc:107] Waiting for socket to connect
0928 02:16:45.365459 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.235.124:34049: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":98}
I20260928 02:16:45.366861 29947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:16:45.366421 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.235.124:34049 (local address 127.0.0.1:38328)
0928 02:16:45.366651 (+ 230us) negotiation.cc:107] Waiting for socket to connect
0928 02:16:45.366741 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.235.124:34049: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":97}
W20260928 02:16:45.367069 29944 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.6.235.124:34049: Network error: Client connection negotiation failed: client connection to 127.6.235.124:34049: connect: Connection refused (error 111)
W20260928 02:16:45.396133 30026 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260928 02:16:45.396569 30026 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:16:45.396705 30026 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:16:45.396845 30026 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260928 02:16:45.399050 30018 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.235.124" port: 34049 } attempt: 4
I20260928 02:16:45.400290 30022 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:16:45.399729 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.235.124:34049 (local address 127.0.0.1:38332)
0928 02:16:45.399979 (+ 250us) negotiation.cc:107] Waiting for socket to connect
0928 02:16:45.400102 (+ 123us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.235.124:34049: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":79}
I20260928 02:16:45.401513 30022 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:16:45.401098 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.235.124:34049 (local address 127.0.0.1:38334)
0928 02:16:45.401291 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0928 02:16:45.401378 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.235.124:34049: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260928 02:16:45.401763 30018 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.6.235.124:34049: Network error: Client connection negotiation failed: client connection to 127.6.235.124:34049: connect: Connection refused (error 111)
W20260928 02:16:45.406431 30026 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 02:16:45.406543 30026 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:16:45.406617 30026 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:16:45.406683 30026 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 02:16:45.406754 30026 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 02:16:45.420323 30026 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.235.84:41181
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790561432251112-7085-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790561432251112-7085-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.6.235.126:44965,127.6.235.125:38915,127.6.235.124:34049
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790561432251112-7085-0/krb5kdc/kudu_127.6.235.124.keytab
--principal=kudu/127.6.235.124
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.235.124:34049
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790561432251112-7085-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.6.235.124
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790561432251112-7085-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 00:51:59 UTC on ccycloud.quasar-axgkwn.root.comops.site
build id 4676464
ASAN enabled
I20260928 02:16:45.421813 30026 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:16:45.423149 30026 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:16:45.435947 30034 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:16:45.436018 30031 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:16:45.436088 30032 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:16:45.437470 30026 server_base.cc:1034] running on GCE node
I20260928 02:16:45.438360 30026 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:16:45.439999 30026 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:16:45.441267 30026 hybrid_clock.cc:648] HybridClock initialized: now 1790561805441172 us; error 60 us; skew 500 ppm
I20260928 02:16:45.441669 30026 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 02:16:45 dist-test-slave-86gv krb5kdc[29864](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561805, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.235.124@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:16:45.445331 30026 init.cc:377] Logged in from keytab as kudu/127.6.235.124@KRBTEST.COM (short username kudu)
I20260928 02:16:45.447085 30026 webserver.cc:460] Webserver started at http://127.6.235.124:38611/ using document root <none> and password file <none>
I20260928 02:16:45.447842 30026 fs_manager.cc:359] Metadata directory not provided
I20260928 02:16:45.447934 30026 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:16:45.448292 30026 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:16:45.451557 30026 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790561432251112-7085-0/minicluster-data/master-2/data/instance:
uuid: "7a99cdfec0b84edeb1508d8fad4d1138"
format_stamp: "Formatted at 2026-09-28 02:16:45 on dist-test-slave-86gv"
server_key: "4181ff99607d4b6a07935a25b2c5bb65"
server_key_iv: "528ca9b62c5a9461e65cb0838b41087f"
server_key_version: "encryptionkey@0"
I20260928 02:16:45.452502 30026 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790561432251112-7085-0/minicluster-data/master-2/wal/instance:
uuid: "7a99cdfec0b84edeb1508d8fad4d1138"
format_stamp: "Formatted at 2026-09-28 02:16:45 on dist-test-slave-86gv"
server_key: "4181ff99607d4b6a07935a25b2c5bb65"
server_key_iv: "528ca9b62c5a9461e65cb0838b41087f"
server_key_version: "encryptionkey@0"
I20260928 02:16:45.458442 30026 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.002s sys 0.006s
I20260928 02:16:45.462496 30041 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:16:45.464217 30026 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260928 02:16:45.464516 30026 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790561432251112-7085-0/minicluster-data/master-2/data,/tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790561432251112-7085-0/minicluster-data/master-2/wal
uuid: "7a99cdfec0b84edeb1508d8fad4d1138"
format_stamp: "Formatted at 2026-09-28 02:16:45 on dist-test-slave-86gv"
server_key: "4181ff99607d4b6a07935a25b2c5bb65"
server_key_iv: "528ca9b62c5a9461e65cb0838b41087f"
server_key_version: "encryptionkey@0"
I20260928 02:16:45.464766 30026 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790561432251112-7085-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790561432251112-7085-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790561432251112-7085-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:16:45.480851 30026 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:16:45.485767 30026 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:16:45.486106 30026 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:16:45.506798 30026 rpc_server.cc:307] RPC server started. Bound to: 127.6.235.124:34049
I20260928 02:16:45.506817 30092 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.235.124:34049 every 8 connection(s)
I20260928 02:16:45.508322 30026 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790561432251112-7085-0/minicluster-data/master-2/data/info.pb
I20260928 02:16:45.510392 7085 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskTlaUuc/build/asan/bin/kudu as pid 30026
I20260928 02:16:45.510617 7085 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskTlaUuc/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790561432251112-7085-0/minicluster-data/master-2/wal/instance
I20260928 02:16:45.511078 7085 external_mini_cluster.cc:1428] Setting key 6babd5b34a5761402db9700f98ef914f
I20260928 02:16:45.513080 30093 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:16:45.521191 30093 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.6.235.126" port: 44965 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 02:16:45 dist-test-slave-86gv krb5kdc[29864](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561803, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.235.124@KRBTEST.COM
Sep 28 02:16:45 dist-test-slave-86gv krb5kdc[29864](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561805, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.235.124@KRBTEST.COM for kudu/127.6.235.126@KRBTEST.COM
I20260928 02:16:45.536268 30096 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:16:45.512964 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38344 (local address 127.6.235.124:34049)
0928 02:16:45.513823 (+ 859us) server_negotiation.cc:206] Beginning negotiation
0928 02:16:45.513835 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0928 02:16:45.513880 (+ 45us) server_negotiation.cc:407] Connection header received
0928 02:16:45.515566 (+ 1686us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:16:45.515590 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:16:45.516066 (+ 476us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:16:45.516552 (+ 486us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:16:45.518111 (+ 1559us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:16:45.519544 (+ 1433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:16:45.520949 (+ 1405us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:16:45.521735 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:16:45.521966 (+ 231us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:16:45.527438 (+ 5472us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:16:45.527447 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:16:45.527460 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:16:45.527500 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:16:45.530712 (+ 3212us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:16:45.531617 (+ 905us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:16:45.531635 (+ 18us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:16:45.531648 (+ 13us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:16:45.531892 (+ 244us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:16:45.532363 (+ 471us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:16:45.532381 (+ 18us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:16:45.532385 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:16:45.533217 (+ 832us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:16:45.533542 (+ 325us) server_negotiation.cc:1035] Waiting for connection context
0928 02:16:45.534068 (+ 526us) server_negotiation.cc:299] Negotiation successful
0928 02:16:45.534604 (+ 536us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":627,"thread_start_us":418,"threads_started":1}
I20260928 02:16:45.541613 30097 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:16:45.524230 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58960 (local address 127.6.235.126:44965)
0928 02:16:45.524864 (+ 634us) server_negotiation.cc:206] Beginning negotiation
0928 02:16:45.524871 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 02:16:45.525352 (+ 481us) server_negotiation.cc:407] Connection header received
0928 02:16:45.525678 (+ 326us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:16:45.525682 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:16:45.525757 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:16:45.525882 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:16:45.527600 (+ 1718us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:16:45.528537 (+ 937us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:16:45.530170 (+ 1633us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:16:45.530584 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:16:45.530727 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:16:45.535019 (+ 4292us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:16:45.535027 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:16:45.535032 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:16:45.535081 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:16:45.538480 (+ 3399us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:16:45.539343 (+ 863us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:16:45.539347 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:16:45.539349 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:16:45.539484 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:16:45.539996 (+ 512us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:16:45.540001 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:16:45.540003 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:16:45.540310 (+ 307us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:16:45.540496 (+ 186us) server_negotiation.cc:1035] Waiting for connection context
0928 02:16:45.541119 (+ 623us) server_negotiation.cc:299] Negotiation successful
0928 02:16:45.541330 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":491,"thread_start_us":302,"threads_started":1}
I20260928 02:16:45.541779 30098 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:16:45.524225 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.235.126:44965 (local address 127.0.0.1:58960)
0928 02:16:45.525189 (+ 964us) negotiation.cc:107] Waiting for socket to connect
0928 02:16:45.525234 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0928 02:16:45.525454 (+ 220us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:16:45.526019 (+ 565us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:16:45.526029 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:16:45.527042 (+ 1013us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:16:45.527431 (+ 389us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:16:45.527441 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:16:45.528722 (+ 1281us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:16:45.528727 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:16:45.529993 (+ 1266us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:16:45.530005 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:16:45.530832 (+ 827us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:16:45.530838 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:16:45.531010 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:16:45.531724 (+ 714us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:16:45.531764 (+ 40us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:16:45.534789 (+ 3025us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:16:45.538659 (+ 3870us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:16:45.538670 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:16:45.538677 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:16:45.539181 (+ 504us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:16:45.539662 (+ 481us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:16:45.539667 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:16:45.539671 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:16:45.539846 (+ 175us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:16:45.540453 (+ 607us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:16:45.540460 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:16:45.540666 (+ 206us) client_negotiation.cc:769] Sending connection context
0928 02:16:45.540938 (+ 272us) client_negotiation.cc:240] Negotiation successful
0928 02:16:45.541328 (+ 390us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":741,"thread_start_us":540,"threads_started":1}
I20260928 02:16:45.543574 30093 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.6.235.125" port: 38915 } has no permanent_uuid. Determining permanent_uuid...
I20260928 02:16:45.544319 29944 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.235.124" port: 34049 } attempt: 4
Sep 28 02:16:45 dist-test-slave-86gv krb5kdc[29864](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561803, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.235.126@KRBTEST.COM
Sep 28 02:16:45 dist-test-slave-86gv krb5kdc[29864](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561804, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.235.126@KRBTEST.COM for kudu/127.6.235.124@KRBTEST.COM
Sep 28 02:16:45 dist-test-slave-86gv krb5kdc[29864](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561805, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.235.124@KRBTEST.COM for kudu/127.6.235.125@KRBTEST.COM
I20260928 02:16:45.558959 30099 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:16:45.539771 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58974 (local address 127.6.235.126:44965)
0928 02:16:45.540658 (+ 887us) server_negotiation.cc:206] Beginning negotiation
0928 02:16:45.540663 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 02:16:45.540681 (+ 18us) server_negotiation.cc:407] Connection header received
0928 02:16:45.540719 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:16:45.540722 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:16:45.540775 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:16:45.540855 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:16:45.542492 (+ 1637us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:16:45.543645 (+ 1153us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:16:45.545381 (+ 1736us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:16:45.546078 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:16:45.546222 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:16:45.551791 (+ 5569us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:16:45.551800 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:16:45.551808 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:16:45.551862 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:16:45.555475 (+ 3613us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:16:45.556564 (+ 1089us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:16:45.556569 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:16:45.556573 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:16:45.556723 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:16:45.557253 (+ 530us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:16:45.557260 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:16:45.557266 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:16:45.557686 (+ 420us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:16:45.557993 (+ 307us) server_negotiation.cc:1035] Waiting for connection context
0928 02:16:45.558325 (+ 332us) server_negotiation.cc:299] Negotiation successful
0928 02:16:45.558546 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":767,"thread_start_us":555,"threads_started":1}
I20260928 02:16:45.560818 30096 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:16:45.544967 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38352 (local address 127.6.235.124:34049)
0928 02:16:45.545123 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0928 02:16:45.545128 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 02:16:45.545278 (+ 150us) server_negotiation.cc:407] Connection header received
0928 02:16:45.545458 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:16:45.545463 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:16:45.545565 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:16:45.545665 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:16:45.546546 (+ 881us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:16:45.547443 (+ 897us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:16:45.548662 (+ 1219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:16:45.549220 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:16:45.549353 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:16:45.553872 (+ 4519us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:16:45.553880 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:16:45.553887 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:16:45.553933 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:16:45.557513 (+ 3580us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:16:45.558370 (+ 857us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:16:45.558376 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:16:45.558379 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:16:45.558567 (+ 188us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:16:45.559247 (+ 680us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:16:45.559255 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:16:45.559258 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:16:45.559601 (+ 343us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:16:45.559754 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0928 02:16:45.560241 (+ 487us) server_negotiation.cc:299] Negotiation successful
0928 02:16:45.560461 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":50,"spinlock_wait_cycles":768}
I20260928 02:16:45.560820 29947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:16:45.544901 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.235.124:34049 (local address 127.0.0.1:38352)
0928 02:16:45.545152 (+ 251us) negotiation.cc:107] Waiting for socket to connect
0928 02:16:45.545170 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0928 02:16:45.545335 (+ 165us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:16:45.545733 (+ 398us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:16:45.545738 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:16:45.546240 (+ 502us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:16:45.546412 (+ 172us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:16:45.546421 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:16:45.547599 (+ 1178us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:16:45.547605 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:16:45.548506 (+ 901us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:16:45.548517 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:16:45.549370 (+ 853us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:16:45.549376 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:16:45.549551 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:16:45.550251 (+ 700us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:16:45.550282 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:16:45.553676 (+ 3394us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:16:45.557680 (+ 4004us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:16:45.557686 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:16:45.557690 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:16:45.558223 (+ 533us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:16:45.558894 (+ 671us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:16:45.558900 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:16:45.558903 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:16:45.559114 (+ 211us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:16:45.559770 (+ 656us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:16:45.559775 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:16:45.559962 (+ 187us) client_negotiation.cc:769] Sending connection context
0928 02:16:45.560141 (+ 179us) client_negotiation.cc:240] Negotiation successful
0928 02:16:45.560433 (+ 292us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":152}
I20260928 02:16:45.560940 30024 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:16:45.545498 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56944 (local address 127.6.235.125:38915)
0928 02:16:45.545744 (+ 246us) server_negotiation.cc:206] Beginning negotiation
0928 02:16:45.545751 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 02:16:45.545769 (+ 18us) server_negotiation.cc:407] Connection header received
0928 02:16:45.545826 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:16:45.545833 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:16:45.545894 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:16:45.546004 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:16:45.546888 (+ 884us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:16:45.547959 (+ 1071us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:16:45.549954 (+ 1995us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:16:45.550449 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:16:45.550607 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:16:45.554474 (+ 3867us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:16:45.554480 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:16:45.554483 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:16:45.554517 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:16:45.557758 (+ 3241us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:16:45.558654 (+ 896us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:16:45.558660 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:16:45.558665 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:16:45.558834 (+ 169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:16:45.559278 (+ 444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:16:45.559284 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:16:45.559289 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:16:45.559613 (+ 324us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:16:45.559746 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0928 02:16:45.560281 (+ 535us) server_negotiation.cc:299] Negotiation successful
0928 02:16:45.560506 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260928 02:16:45.561849 30098 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:16:45.544497 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.235.125:38915 (local address 127.0.0.1:56944)
0928 02:16:45.544764 (+ 267us) negotiation.cc:107] Waiting for socket to connect
0928 02:16:45.544782 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0928 02:16:45.544877 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:16:45.546024 (+ 1147us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:16:45.546028 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:16:45.546502 (+ 474us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:16:45.546715 (+ 213us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:16:45.546729 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:16:45.548153 (+ 1424us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:16:45.548160 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:16:45.549790 (+ 1630us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:16:45.549808 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:16:45.550619 (+ 811us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:16:45.550625 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:16:45.550813 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:16:45.551366 (+ 553us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:16:45.551383 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:16:45.554273 (+ 2890us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:16:45.557923 (+ 3650us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:16:45.557928 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:16:45.557932 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:16:45.558506 (+ 574us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:16:45.558983 (+ 477us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:16:45.558987 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:16:45.558992 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:16:45.559155 (+ 163us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:16:45.559824 (+ 669us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:16:45.559827 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:16:45.559980 (+ 153us) client_negotiation.cc:769] Sending connection context
0928 02:16:45.561320 (+ 1340us) client_negotiation.cc:240] Negotiation successful
0928 02:16:45.561621 (+ 301us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":160,"spinlock_wait_cycles":32512}
I20260928 02:16:45.562844 30093 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.6.235.124" port: 34049 } has no permanent_uuid. Determining permanent_uuid...
I20260928 02:16:45.567838 29944 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 69d84461ef5b4b29aaf2cd81fd158769: Bootstrap starting.
Sep 28 02:16:45 dist-test-slave-86gv krb5kdc[29864](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561805, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.235.124@KRBTEST.COM for kudu/127.6.235.124@KRBTEST.COM
I20260928 02:16:45.572252 29944 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 69d84461ef5b4b29aaf2cd81fd158769: Neither blocks nor log segments found. Creating new log.
Sep 28 02:16:45 dist-test-slave-86gv krb5kdc[29864](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561803, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.235.125@KRBTEST.COM
I20260928 02:16:45.573523 29944 log.cc:824] T 00000000000000000000000000000000 P 69d84461ef5b4b29aaf2cd81fd158769: Log is configured to *not* fsync() on all Append() calls
I20260928 02:16:45.576881 29944 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 69d84461ef5b4b29aaf2cd81fd158769: No bootstrap required, opened a new log
I20260928 02:16:45.578061 30098 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:16:45.563550 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.235.124:34049 (local address 127.0.0.1:38362)
0928 02:16:45.563821 (+ 271us) negotiation.cc:107] Waiting for socket to connect
0928 02:16:45.563840 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0928 02:16:45.563933 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:16:45.564260 (+ 327us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:16:45.564265 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:16:45.564708 (+ 443us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:16:45.564920 (+ 212us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:16:45.564932 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:16:45.566274 (+ 1342us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:16:45.566280 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:16:45.567442 (+ 1162us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:16:45.567453 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:16:45.568509 (+ 1056us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:16:45.568513 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:16:45.568669 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:16:45.569324 (+ 655us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:16:45.569341 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:16:45.571483 (+ 2142us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:16:45.575379 (+ 3896us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:16:45.575384 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:16:45.575388 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:16:45.575919 (+ 531us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:16:45.576314 (+ 395us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:16:45.576321 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:16:45.576326 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:16:45.576474 (+ 148us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:16:45.577219 (+ 745us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:16:45.577223 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:16:45.577383 (+ 160us) client_negotiation.cc:769] Sending connection context
0928 02:16:45.577548 (+ 165us) client_negotiation.cc:240] Negotiation successful
0928 02:16:45.577809 (+ 261us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":148}
I20260928 02:16:45.578933 30096 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:16:45.563802 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38362 (local address 127.6.235.124:34049)
0928 02:16:45.563984 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0928 02:16:45.563989 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 02:16:45.564004 (+ 15us) server_negotiation.cc:407] Connection header received
0928 02:16:45.564048 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:16:45.564054 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:16:45.564124 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:16:45.564217 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:16:45.565056 (+ 839us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:16:45.566110 (+ 1054us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:16:45.567716 (+ 1606us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:16:45.568333 (+ 617us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:16:45.568453 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:16:45.571684 (+ 3231us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:16:45.571689 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:16:45.571692 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:16:45.571752 (+ 60us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:16:45.575219 (+ 3467us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:16:45.576049 (+ 830us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:16:45.576053 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:16:45.576057 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:16:45.576195 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:16:45.576601 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:16:45.576608 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:16:45.576613 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:16:45.577011 (+ 398us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:16:45.577168 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0928 02:16:45.578372 (+ 1204us) server_negotiation.cc:299] Negotiation successful
0928 02:16:45.578580 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260928 02:16:45.581074 30024 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:16:45.561127 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56958 (local address 127.6.235.125:38915)
0928 02:16:45.561276 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0928 02:16:45.561281 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 02:16:45.561394 (+ 113us) server_negotiation.cc:407] Connection header received
0928 02:16:45.561884 (+ 490us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:16:45.561891 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:16:45.561974 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:16:45.562087 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:16:45.563991 (+ 1904us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:16:45.564874 (+ 883us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:16:45.566117 (+ 1243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:16:45.566770 (+ 653us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:16:45.566900 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:16:45.573683 (+ 6783us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:16:45.573691 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:16:45.573695 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:16:45.573736 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:16:45.577392 (+ 3656us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:16:45.578578 (+ 1186us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:16:45.578584 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:16:45.578587 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:16:45.578747 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:16:45.579335 (+ 588us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:16:45.579341 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:16:45.579344 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:16:45.579730 (+ 386us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:16:45.580022 (+ 292us) server_negotiation.cc:1035] Waiting for connection context
0928 02:16:45.580407 (+ 385us) server_negotiation.cc:299] Negotiation successful
0928 02:16:45.580660 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":17,"server-negotiator.queue_time_us":63}
I20260928 02:16:45.584331 30093 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7a99cdfec0b84edeb1508d8fad4d1138: Bootstrap starting.
I20260928 02:16:45.586628 7085 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 02:16:45.587538 29944 raft_consensus.cc:348] T 00000000000000000000000000000000 P 69d84461ef5b4b29aaf2cd81fd158769 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "69d84461ef5b4b29aaf2cd81fd158769" member_type: VOTER last_known_addr { host: "127.6.235.126" port: 44965 } } peers { permanent_uuid: "387b63f25d984ebfa73f753ff65b2d8f" member_type: VOTER last_known_addr { host: "127.6.235.125" port: 38915 } } peers { permanent_uuid: "7a99cdfec0b84edeb1508d8fad4d1138" member_type: VOTER last_known_addr { host: "127.6.235.124" port: 34049 } }
I20260928 02:16:45.587888 29944 raft_consensus.cc:729] T 00000000000000000000000000000000 P 69d84461ef5b4b29aaf2cd81fd158769 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 69d84461ef5b4b29aaf2cd81fd158769, State: Initialized, Role: FOLLOWER
I20260928 02:16:45.588280 30093 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7a99cdfec0b84edeb1508d8fad4d1138: Neither blocks nor log segments found. Creating new log.
I20260928 02:16:45.588333 29944 consensus_queue.cc:260] T 00000000000000000000000000000000 P 69d84461ef5b4b29aaf2cd81fd158769 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "69d84461ef5b4b29aaf2cd81fd158769" member_type: VOTER last_known_addr { host: "127.6.235.126" port: 44965 } } peers { permanent_uuid: "387b63f25d984ebfa73f753ff65b2d8f" member_type: VOTER last_known_addr { host: "127.6.235.125" port: 38915 } } peers { permanent_uuid: "7a99cdfec0b84edeb1508d8fad4d1138" member_type: VOTER last_known_addr { host: "127.6.235.124" port: 34049 } }
I20260928 02:16:45.589368 30101 sys_catalog.cc:455] T 00000000000000000000000000000000 P 69d84461ef5b4b29aaf2cd81fd158769 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "69d84461ef5b4b29aaf2cd81fd158769" member_type: VOTER last_known_addr { host: "127.6.235.126" port: 44965 } } peers { permanent_uuid: "387b63f25d984ebfa73f753ff65b2d8f" member_type: VOTER last_known_addr { host: "127.6.235.125" port: 38915 } } peers { permanent_uuid: "7a99cdfec0b84edeb1508d8fad4d1138" member_type: VOTER last_known_addr { host: "127.6.235.124" port: 34049 } } }
I20260928 02:16:45.589695 30093 log.cc:824] T 00000000000000000000000000000000 P 7a99cdfec0b84edeb1508d8fad4d1138: Log is configured to *not* fsync() on all Append() calls
I20260928 02:16:45.589922 30101 sys_catalog.cc:458] T 00000000000000000000000000000000 P 69d84461ef5b4b29aaf2cd81fd158769 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 02:16:45.589764 29919 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "69d84461ef5b4b29aaf2cd81fd158769"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58974
I20260928 02:16:45.590924 29944 sys_catalog.cc:564] T 00000000000000000000000000000000 P 69d84461ef5b4b29aaf2cd81fd158769 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 02:16:45.590981 29919 raft_consensus.cc:482] T 00000000000000000000000000000000 P 69d84461ef5b4b29aaf2cd81fd158769 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260928 02:16:45.591239 29919 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 69d84461ef5b4b29aaf2cd81fd158769 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:16:45.593293 30093 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7a99cdfec0b84edeb1508d8fad4d1138: No bootstrap required, opened a new log
I20260928 02:16:45.594313 29919 raft_consensus.cc:504] T 00000000000000000000000000000000 P 69d84461ef5b4b29aaf2cd81fd158769 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "69d84461ef5b4b29aaf2cd81fd158769" member_type: VOTER last_known_addr { host: "127.6.235.126" port: 44965 } } peers { permanent_uuid: "387b63f25d984ebfa73f753ff65b2d8f" member_type: VOTER last_known_addr { host: "127.6.235.125" port: 38915 } } peers { permanent_uuid: "7a99cdfec0b84edeb1508d8fad4d1138" member_type: VOTER last_known_addr { host: "127.6.235.124" port: 34049 } }
I20260928 02:16:45.596505 29919 leader_election.cc:288] T 00000000000000000000000000000000 P 69d84461ef5b4b29aaf2cd81fd158769 [CANDIDATE]: Term 1 election: Requested vote from peers 387b63f25d984ebfa73f753ff65b2d8f (127.6.235.125:38915), 7a99cdfec0b84edeb1508d8fad4d1138 (127.6.235.124:34049)
I20260928 02:16:45.608400 30093 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7a99cdfec0b84edeb1508d8fad4d1138 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "69d84461ef5b4b29aaf2cd81fd158769" member_type: VOTER last_known_addr { host: "127.6.235.126" port: 44965 } } peers { permanent_uuid: "387b63f25d984ebfa73f753ff65b2d8f" member_type: VOTER last_known_addr { host: "127.6.235.125" port: 38915 } } peers { permanent_uuid: "7a99cdfec0b84edeb1508d8fad4d1138" member_type: VOTER last_known_addr { host: "127.6.235.124" port: 34049 } }
I20260928 02:16:45.609078 30093 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7a99cdfec0b84edeb1508d8fad4d1138 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a99cdfec0b84edeb1508d8fad4d1138, State: Initialized, Role: FOLLOWER
I20260928 02:16:45.609681 30093 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7a99cdfec0b84edeb1508d8fad4d1138 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "69d84461ef5b4b29aaf2cd81fd158769" member_type: VOTER last_known_addr { host: "127.6.235.126" port: 44965 } } peers { permanent_uuid: "387b63f25d984ebfa73f753ff65b2d8f" member_type: VOTER last_known_addr { host: "127.6.235.125" port: 38915 } } peers { permanent_uuid: "7a99cdfec0b84edeb1508d8fad4d1138" member_type: VOTER last_known_addr { host: "127.6.235.124" port: 34049 } }
I20260928 02:16:45.617194 30117 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7a99cdfec0b84edeb1508d8fad4d1138 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "69d84461ef5b4b29aaf2cd81fd158769" member_type: VOTER last_known_addr { host: "127.6.235.126" port: 44965 } } peers { permanent_uuid: "387b63f25d984ebfa73f753ff65b2d8f" member_type: VOTER last_known_addr { host: "127.6.235.125" port: 38915 } } peers { permanent_uuid: "7a99cdfec0b84edeb1508d8fad4d1138" member_type: VOTER last_known_addr { host: "127.6.235.124" port: 34049 } } }
I20260928 02:16:45.617722 30117 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7a99cdfec0b84edeb1508d8fad4d1138 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 02:16:45.618623 30093 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7a99cdfec0b84edeb1508d8fad4d1138 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 02:16:45.618716 30068 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "69d84461ef5b4b29aaf2cd81fd158769" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "7a99cdfec0b84edeb1508d8fad4d1138"
I20260928 02:16:45.619367 30068 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7a99cdfec0b84edeb1508d8fad4d1138 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:16:45.625946 30068 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 7a99cdfec0b84edeb1508d8fad4d1138 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 69d84461ef5b4b29aaf2cd81fd158769 in term 1.
I20260928 02:16:45.628863 29893 leader_election.cc:302] T 00000000000000000000000000000000 P 69d84461ef5b4b29aaf2cd81fd158769 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 69d84461ef5b4b29aaf2cd81fd158769, 7a99cdfec0b84edeb1508d8fad4d1138; no voters:
I20260928 02:16:45.629565 30101 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 69d84461ef5b4b29aaf2cd81fd158769 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:16:45.615777 29877 master_runner.cc:186] Error getting master registration for 127.6.235.125:38915: OK, instance_id { permanent_uuid: "387b63f25d984ebfa73f753ff65b2d8f" instance_seqno: 1790561804962154 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
W20260928 02:16:45.637692 29898 tablet.cc:2369] T 00000000000000000000000000000000 P 69d84461ef5b4b29aaf2cd81fd158769: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 02:16:45.638670 29877 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260928 02:16:45.639168 30101 raft_consensus.cc:686] T 00000000000000000000000000000000 P 69d84461ef5b4b29aaf2cd81fd158769 [term 1 LEADER]: Becoming Leader. State: Replica: 69d84461ef5b4b29aaf2cd81fd158769, State: Running, Role: LEADER
I20260928 02:16:45.639816 30101 consensus_queue.cc:237] T 00000000000000000000000000000000 P 69d84461ef5b4b29aaf2cd81fd158769 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "69d84461ef5b4b29aaf2cd81fd158769" member_type: VOTER last_known_addr { host: "127.6.235.126" port: 44965 } } peers { permanent_uuid: "387b63f25d984ebfa73f753ff65b2d8f" member_type: VOTER last_known_addr { host: "127.6.235.125" port: 38915 } } peers { permanent_uuid: "7a99cdfec0b84edeb1508d8fad4d1138" member_type: VOTER last_known_addr { host: "127.6.235.124" port: 34049 } }
I20260928 02:16:45.644220 30123 sys_catalog.cc:455] T 00000000000000000000000000000000 P 69d84461ef5b4b29aaf2cd81fd158769 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 69d84461ef5b4b29aaf2cd81fd158769. Latest consensus state: current_term: 1 leader_uuid: "69d84461ef5b4b29aaf2cd81fd158769" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "69d84461ef5b4b29aaf2cd81fd158769" member_type: VOTER last_known_addr { host: "127.6.235.126" port: 44965 } } peers { permanent_uuid: "387b63f25d984ebfa73f753ff65b2d8f" member_type: VOTER last_known_addr { host: "127.6.235.125" port: 38915 } } peers { permanent_uuid: "7a99cdfec0b84edeb1508d8fad4d1138" member_type: VOTER last_known_addr { host: "127.6.235.124" port: 34049 } } }
I20260928 02:16:45.644812 30123 sys_catalog.cc:458] T 00000000000000000000000000000000 P 69d84461ef5b4b29aaf2cd81fd158769 [sys.catalog]: This master's current role is: LEADER
W20260928 02:16:45.646662 30115 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 69d84461ef5b4b29aaf2cd81fd158769: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 02:16:45.646870 30115 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 02:16:45.659809 30128 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 02:16:45.662357 30128 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 02:16:45.664517 30099 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:16:45.623381 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58980 (local address 127.6.235.126:44965)
0928 02:16:45.623880 (+ 499us) server_negotiation.cc:206] Beginning negotiation
0928 02:16:45.623888 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 02:16:45.640195 (+ 16307us) server_negotiation.cc:407] Connection header received
0928 02:16:45.641141 (+ 946us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:16:45.641151 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:16:45.641267 (+ 116us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:16:45.641362 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:16:45.645398 (+ 4036us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:16:45.646501 (+ 1103us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:16:45.648909 (+ 2408us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:16:45.649691 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:16:45.649804 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:16:45.654221 (+ 4417us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:16:45.654231 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:16:45.654238 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:16:45.654288 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:16:45.659151 (+ 4863us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:16:45.661102 (+ 1951us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:16:45.661109 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:16:45.661112 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:16:45.661305 (+ 193us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:16:45.662151 (+ 846us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:16:45.662159 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:16:45.662165 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:16:45.662535 (+ 370us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:16:45.662794 (+ 259us) server_negotiation.cc:1035] Waiting for connection context
0928 02:16:45.663894 (+ 1100us) server_negotiation.cc:299] Negotiation successful
0928 02:16:45.664161 (+ 267us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":315}
I20260928 02:16:45.668820 30024 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:16:45.623115 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56966 (local address 127.6.235.125:38915)
0928 02:16:45.623654 (+ 539us) server_negotiation.cc:206] Beginning negotiation
0928 02:16:45.623664 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 02:16:45.641006 (+ 17342us) server_negotiation.cc:407] Connection header received
0928 02:16:45.641839 (+ 833us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:16:45.641847 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:16:45.641940 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:16:45.642015 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:16:45.650750 (+ 8735us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:16:45.651840 (+ 1090us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:16:45.654320 (+ 2480us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:16:45.655043 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:16:45.655155 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:16:45.659932 (+ 4777us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:16:45.659940 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:16:45.659946 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:16:45.659990 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:16:45.663747 (+ 3757us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:16:45.665491 (+ 1744us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:16:45.665498 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:16:45.665503 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:16:45.665715 (+ 212us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:16:45.666413 (+ 698us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:16:45.666419 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:16:45.666424 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:16:45.666864 (+ 440us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:16:45.667123 (+ 259us) server_negotiation.cc:1035] Waiting for connection context
0928 02:16:45.668020 (+ 897us) server_negotiation.cc:299] Negotiation successful
0928 02:16:45.668270 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":341}
I20260928 02:16:45.677361 30068 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 7a99cdfec0b84edeb1508d8fad4d1138 [term 1 FOLLOWER]: Refusing update from remote peer 69d84461ef5b4b29aaf2cd81fd158769: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 02:16:45.679085 30123 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 69d84461ef5b4b29aaf2cd81fd158769 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a99cdfec0b84edeb1508d8fad4d1138" member_type: VOTER last_known_addr { host: "127.6.235.124" port: 34049 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 02:16:45.680462 30096 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:16:45.639915 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38378 (local address 127.6.235.124:34049)
0928 02:16:45.640121 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0928 02:16:45.640127 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 02:16:45.647594 (+ 7467us) server_negotiation.cc:407] Connection header received
0928 02:16:45.647937 (+ 343us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:16:45.647943 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:16:45.648050 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:16:45.648120 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:16:45.660095 (+ 11975us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:16:45.661265 (+ 1170us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:16:45.663498 (+ 2233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:16:45.664212 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:16:45.664313 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:16:45.669330 (+ 5017us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:16:45.669338 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:16:45.669345 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:16:45.669389 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:16:45.675520 (+ 6131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:16:45.676917 (+ 1397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:16:45.676925 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:16:45.676929 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:16:45.677088 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:16:45.678010 (+ 922us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:16:45.678030 (+ 20us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:16:45.678034 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:16:45.678604 (+ 570us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:16:45.678868 (+ 264us) server_negotiation.cc:1035] Waiting for connection context
0928 02:16:45.679803 (+ 935us) server_negotiation.cc:299] Negotiation successful
0928 02:16:45.680061 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260928 02:16:45.698257 30117 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7a99cdfec0b84edeb1508d8fad4d1138 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 69d84461ef5b4b29aaf2cd81fd158769. Latest consensus state: current_term: 1 leader_uuid: "69d84461ef5b4b29aaf2cd81fd158769" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "69d84461ef5b4b29aaf2cd81fd158769" member_type: VOTER last_known_addr { host: "127.6.235.126" port: 44965 } } peers { permanent_uuid: "387b63f25d984ebfa73f753ff65b2d8f" member_type: VOTER last_known_addr { host: "127.6.235.125" port: 38915 } } peers { permanent_uuid: "7a99cdfec0b84edeb1508d8fad4d1138" member_type: VOTER last_known_addr { host: "127.6.235.124" port: 34049 } } }
I20260928 02:16:45.698622 30117 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7a99cdfec0b84edeb1508d8fad4d1138 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 02:16:45.699473 30101 sys_catalog.cc:455] T 00000000000000000000000000000000 P 69d84461ef5b4b29aaf2cd81fd158769 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "69d84461ef5b4b29aaf2cd81fd158769" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "69d84461ef5b4b29aaf2cd81fd158769" member_type: VOTER last_known_addr { host: "127.6.235.126" port: 44965 } } peers { permanent_uuid: "387b63f25d984ebfa73f753ff65b2d8f" member_type: VOTER last_known_addr { host: "127.6.235.125" port: 38915 } } peers { permanent_uuid: "7a99cdfec0b84edeb1508d8fad4d1138" member_type: VOTER last_known_addr { host: "127.6.235.124" port: 34049 } } }
I20260928 02:16:45.699832 30101 sys_catalog.cc:458] T 00000000000000000000000000000000 P 69d84461ef5b4b29aaf2cd81fd158769 [sys.catalog]: This master's current role is: LEADER
I20260928 02:16:45.700313 30133 mvcc.cc:205] Tried to move back new op lower bound from 7334141156039864320 to 7334141155907162112. Current Snapshot: MvccSnapshot[applied={T|T < 7334141156039864320}]
I20260928 02:16:45.701865 30018 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.235.124" port: 34049 } attempt: 5
I20260928 02:16:45.702087 30131 mvcc.cc:205] Tried to move back new op lower bound from 7334141156039864320 to 7334141155907162112. Current Snapshot: MvccSnapshot[applied={T|T < 7334141156039864320}]
I20260928 02:16:45.709419 30117 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7a99cdfec0b84edeb1508d8fad4d1138 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "69d84461ef5b4b29aaf2cd81fd158769" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "69d84461ef5b4b29aaf2cd81fd158769" member_type: VOTER last_known_addr { host: "127.6.235.126" port: 44965 } } peers { permanent_uuid: "387b63f25d984ebfa73f753ff65b2d8f" member_type: VOTER last_known_addr { host: "127.6.235.125" port: 38915 } } peers { permanent_uuid: "7a99cdfec0b84edeb1508d8fad4d1138" member_type: VOTER last_known_addr { host: "127.6.235.124" port: 34049 } } }
I20260928 02:16:45.709833 30117 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7a99cdfec0b84edeb1508d8fad4d1138 [sys.catalog]: This master's current role is: FOLLOWER
Sep 28 02:16:45 dist-test-slave-86gv krb5kdc[29864](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790561804, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.235.125@KRBTEST.COM for kudu/127.6.235.124@KRBTEST.COM
I20260928 02:16:45.718623 30128 catalog_manager.cc:1347] Generated new cluster ID: 662459418cb44231bd639e838dc93d9a
I20260928 02:16:45.718746 30128 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 02:16:45.728863 30022 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:16:45.702704 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.235.124:34049 (local address 127.0.0.1:38394)
0928 02:16:45.703070 (+ 366us) negotiation.cc:107] Waiting for socket to connect
0928 02:16:45.703090 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0928 02:16:45.703230 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:16:45.704827 (+ 1597us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:16:45.704833 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:16:45.705439 (+ 606us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:16:45.705682 (+ 243us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:16:45.705697 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:16:45.707716 (+ 2019us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:16:45.707722 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:16:45.709185 (+ 1463us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:16:45.709203 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:16:45.714142 (+ 4939us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:16:45.714151 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:16:45.714435 (+ 284us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:16:45.715255 (+ 820us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:16:45.715284 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:16:45.719162 (+ 3878us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:16:45.724974 (+ 5812us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:16:45.724984 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:16:45.724990 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:16:45.725754 (+ 764us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:16:45.726314 (+ 560us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:16:45.726322 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:16:45.726329 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:16:45.726552 (+ 223us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:16:45.727515 (+ 963us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:16:45.727523 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:16:45.727777 (+ 254us) client_negotiation.cc:769] Sending connection context
0928 02:16:45.727952 (+ 175us) client_negotiation.cc:240] Negotiation successful
0928 02:16:45.728292 (+ 340us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":224}
I20260928 02:16:45.730263 30096 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:16:45.703460 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38394 (local address 127.6.235.124:34049)
0928 02:16:45.703663 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0928 02:16:45.703669 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 02:16:45.703687 (+ 18us) server_negotiation.cc:407] Connection header received
0928 02:16:45.703746 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:16:45.703764 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:16:45.703839 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:16:45.703934 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:16:45.706336 (+ 2402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:16:45.707543 (+ 1207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:16:45.709432 (+ 1889us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:16:45.710229 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:16:45.710371 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:16:45.719496 (+ 9125us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:16:45.719506 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:16:45.719511 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:16:45.719594 (+ 83us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:16:45.724633 (+ 5039us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:16:45.725951 (+ 1318us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:16:45.725960 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:16:45.725967 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:16:45.726154 (+ 187us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:16:45.726725 (+ 571us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:16:45.726733 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:16:45.726739 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:16:45.727241 (+ 502us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:16:45.727435 (+ 194us) server_negotiation.cc:1035] Waiting for connection context
0928 02:16:45.729311 (+ 1876us) server_negotiation.cc:299] Negotiation successful
0928 02:16:45.729663 (+ 352us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260928 02:16:45.740061 30141 catalog_manager.cc:1259] Loaded cluster ID: 662459418cb44231bd639e838dc93d9a
I20260928 02:16:45.740190 30141 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 7a99cdfec0b84edeb1508d8fad4d1138: loading cluster ID for follower catalog manager: success
W20260928 02:16:45.741819 30141 catalog_manager.cc:1576] T 00000000000000000000000000000000 P 7a99cdfec0b84edeb1508d8fad4d1138: acquiring CA information for follower catalog manager: Not found: root CA entry not found
W20260928 02:16:45.742399 30141 catalog_manager.cc:873] Not found: root CA entry not found: failed to prepare follower catalog manager, will retry
I20260928 02:16:45.743276 30018 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 387b63f25d984ebfa73f753ff65b2d8f: Bootstrap starting.
I20260928 02:16:45.747872 30018 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 387b63f25d984ebfa73f753ff65b2d8f: Neither blocks nor log segments found. Creating new log.
I20260928 02:16:45.755743 30018 log.cc:824] T 00000000000000000000000000000000 P 387b63f25d984ebfa73f753ff65b2d8f: Log is configured to *not* fsync() on all Append() calls
I20260928 02:16:45.762573 30128 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 02:16:45.764673 30128 catalog_manager.cc:1504] Loading token signing keys...
I20260928 02:16:45.748857 30026 master_runner.cc:186] Error getting master registration for 127.6.235.125:38915: OK, instance_id { permanent_uuid: "387b63f25d984ebfa73f753ff65b2d8f" instance_seqno: 1790561804962154 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260928 02:16:45.794983 30026 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260928 02:16:45.806052 30018 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 387b63f25d984ebfa73f753ff65b2d8f: No bootstrap required, opened a new log
I20260928 02:16:45.818786 30128 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 69d84461ef5b4b29aaf2cd81fd158769: Generated new TSK 0
I20260928 02:16:45.820109 30128 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 02:16:45.820757 30018 raft_consensus.cc:348] T 00000000000000000000000000000000 P 387b63f25d984ebfa73f753ff65b2d8f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "69d84461ef5b4b29aaf2cd81fd158769" member_type: VOTER last_known_addr { host: "127.6.235.126" port: 44965 } } peers { permanent_uuid: "387b63f25d984ebfa73f753ff65b2d8f" member_type: VOTER last_known_addr { host: "127.6.235.125" port: 38915 } } peers { permanent_uuid: "7a99cdfec0b84edeb1508d8fad4d1138" member_type: VOTER last_known_addr { host: "127.6.235.124" port: 34049 } }
I20260928 02:16:45.821260 30018 raft_consensus.cc:729] T 00000000000000000000000000000000 P 387b63f25d984ebfa73f753ff65b2d8f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 387b63f25d984ebfa73f753ff65b2d8f, State: Initialized, Role: FOLLOWER
I20260928 02:16:45.821871 30018 consensus_queue.cc:260] T 00000000000000000000000000000000 P 387b63f25d984ebfa73f753ff65b2d8f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "69d84461ef5b4b29aaf2cd81fd158769" member_type: VOTER last_known_addr { host: "127.6.235.126" port: 44965 } } peers { permanent_uuid: "387b63f25d984ebfa73f753ff65b2d8f" member_type: VOTER last_known_addr { host: "127.6.235.125" port: 38915 } } peers { permanent_uuid: "7a99cdfec0b84edeb1508d8fad4d1138" member_type: VOTER last_known_addr { host: "127.6.235.124" port: 34049 } }
I20260928 02:16:45.837553 30144 sys_catalog.cc:455] T 00000000000000000000000000000000 P 387b63f25d984ebfa73f753ff65b2d8f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "69d84461ef5b4b29aaf2cd81fd158769" member_type: VOTER last_known_addr { host: "127.6.235.126" port: 44965 } } peers { permanent_uuid: "387b63f25d984ebfa73f753ff65b2d8f" member_type: VOTER last_known_addr { host: "127.6.235.125" port: 38915 } } peers { permanent_uuid: "7a99cdfec0b84edeb1508d8fad4d1138" member_type: VOTER last_known_addr { host: "127.6.235.124" port: 34049 } } }
I20260928 02:16:45.838035 30144 sys_catalog.cc:458] T 00000000000000000000000000000000 P 387b63f25d984ebfa73f753ff65b2d8f [sys.catalog]: This master's current role is: FOLLOWER
I20260928 02:16:45.841161 30018 sys_catalog.cc:564] T 00000000000000000000000000000000 P 387b63f25d984ebfa73f753ff65b2d8f [sys.catalog]: configured and running, proceeding with master startup.
I20260928 02:16:45.845239 29993 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "69d84461ef5b4b29aaf2cd81fd158769" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "387b63f25d984ebfa73f753ff65b2d8f"
I20260928 02:16:45.845978 29993 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 387b63f25d984ebfa73f753ff65b2d8f [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:16:45.854468 29993 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 387b63f25d984ebfa73f753ff65b2d8f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 69d84461ef5b4b29aaf2cd81fd158769 in term 1.
I20260928 02:16:45.855625 29992 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 387b63f25d984ebfa73f753ff65b2d8f [term 1 FOLLOWER]: Refusing update from remote peer 69d84461ef5b4b29aaf2cd81fd158769: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 02:16:45.858240 30101 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 69d84461ef5b4b29aaf2cd81fd158769 [LEADER]: Connected to new peer: Peer: permanent_uuid: "387b63f25d984ebfa73f753ff65b2d8f" member_type: VOTER last_known_addr { host: "127.6.235.125" port: 38915 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 02:16:45.909895 30144 sys_catalog.cc:455] T 00000000000000000000000000000000 P 387b63f25d984ebfa73f753ff65b2d8f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 69d84461ef5b4b29aaf2cd81fd158769. Latest consensus state: current_term: 1 leader_uuid: "69d84461ef5b4b29aaf2cd81fd158769" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "69d84461ef5b4b29aaf2cd81fd158769" member_type: VOTER last_known_addr { host: "127.6.235.126" port: 44965 } } peers { permanent_uuid: "387b63f25d984ebfa73f753ff65b2d8f" member_type: VOTER last_known_addr { host: "127.6.235.125" port: 38915 } } peers { permanent_uuid: "7a99cdfec0b84edeb1508d8fad4d1138" member_type: VOTER last_known_addr { host: "127.6.235.124" port: 34049 } } }
I20260928 02:16:45.909827 30152 sys_catalog.cc:455] T 00000000000000000000000000000000 P 387b63f25d984ebfa73f753ff65b2d8f [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "69d84461ef5b4b29aaf2cd81fd158769" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "69d84461ef5b4b29aaf2cd81fd158769" member_type: VOTER last_known_addr { host: "127.6.235.126" port: 44965 } } peers { permanent_uuid: "387b63f25d984ebfa73f753ff65b2d8f" member_type: VOTER last_known_addr { host: "127.6.235.125" port: 38915 } } peers { permanent_uuid: "7a99cdfec0b84edeb1508d8fad4d1138" member_type: VOTER last_known_addr { host: "127.6.235.124" port: 34049 } } }
I20260928 02:16:45.910332 30144 sys_catalog.cc:458] T 00000000000000000000000000000000 P 387b63f25d984ebfa73f753ff65b2d8f [sys.catalog]: This master's current role is: FOLLOWER
I20260928 02:16:45.910359 30152 sys_catalog.cc:458] T 00000000000000000000000000000000 P 387b63f25d984ebfa73f753ff65b2d8f [sys.catalog]: This master's current role is: FOLLOWER
I20260928 02:16:45.938846 30163 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
W20260928 02:16:45.965994 30099 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:16:45.958614 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58986 (local address 127.6.235.126:44965)
0928 02:16:45.958842 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0928 02:16:45.958849 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 02:16:45.963053 (+ 4204us) server_negotiation.cc:407] Connection header received
0928 02:16:45.963717 (+ 664us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:16:45.963727 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:16:45.963847 (+ 120us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:16:45.963944 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:16:45.965546 (+ 1602us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:58986: BlockingRecv error: recv got EOF from 127.0.0.1:58986 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
W20260928 02:16:45.970021 30024 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:16:45.958667 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56974 (local address 127.6.235.125:38915)
0928 02:16:45.958871 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0928 02:16:45.958878 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 02:16:45.967125 (+ 8247us) server_negotiation.cc:407] Connection header received
0928 02:16:45.967511 (+ 386us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:16:45.967522 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:16:45.967642 (+ 120us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:16:45.967740 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:16:45.969833 (+ 2093us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56974: BlockingRecv error: recv got EOF from 127.0.0.1:56974 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
W20260928 02:16:45.972278 30096 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:16:45.969314 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38402 (local address 127.6.235.124:34049)
0928 02:16:45.969528 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0928 02:16:45.969536 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 02:16:45.970679 (+ 1143us) server_negotiation.cc:407] Connection header received
0928 02:16:45.970895 (+ 216us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:16:45.970900 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:16:45.970985 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:16:45.971098 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:16:45.972050 (+ 952us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38402: BlockingRecv error: recv got EOF from 127.0.0.1:38402 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
W20260928 02:16:45.978749 30024 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:16:45.976347 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56984 (local address 127.6.235.125:38915)
0928 02:16:45.976538 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0928 02:16:45.976543 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 02:16:45.977291 (+ 748us) server_negotiation.cc:407] Connection header received
0928 02:16:45.977540 (+ 249us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:16:45.977544 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:16:45.977643 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:16:45.977767 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:16:45.978478 (+ 711us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56984: BlockingRecv error: recv got EOF from 127.0.0.1:56984 (error 108)
Metrics: {"server-negotiator.queue_time_us":73}
W20260928 02:16:45.985392 30024 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:16:45.983235 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57000 (local address 127.6.235.125:38915)
0928 02:16:45.983462 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0928 02:16:45.983472 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 02:16:45.984007 (+ 535us) server_negotiation.cc:407] Connection header received
0928 02:16:45.984221 (+ 214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:16:45.984231 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:16:45.984304 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:16:45.984420 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:16:45.985178 (+ 758us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57000: BlockingRecv error: recv got EOF from 127.0.0.1:57000 (error 108)
Metrics: {"server-negotiator.queue_time_us":75}
W20260928 02:16:46.030911 30024 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:16:46.022545 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57002 (local address 127.6.235.125:38915)
0928 02:16:46.022782 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0928 02:16:46.022792 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 02:16:46.029197 (+ 6405us) server_negotiation.cc:407] Connection header received
0928 02:16:46.029479 (+ 282us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:16:46.029483 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:16:46.029578 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:16:46.029643 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:16:46.030609 (+ 966us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57002: BlockingRecv error: recv got EOF from 127.0.0.1:57002 (error 108)
Metrics: {"server-negotiator.queue_time_us":73}
W20260928 02:16:46.041601 30024 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:16:46.039274 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57012 (local address 127.6.235.125:38915)
0928 02:16:46.039515 (+ 241us) server_negotiation.cc:206] Beginning negotiation
0928 02:16:46.039523 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 02:16:46.039957 (+ 434us) server_negotiation.cc:407] Connection header received
0928 02:16:46.040227 (+ 270us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:16:46.040232 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:16:46.040326 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:16:46.040456 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:16:46.041347 (+ 891us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57012: BlockingRecv error: recv got EOF from 127.0.0.1:57012 (error 108)
Metrics: {"server-negotiator.queue_time_us":87}
W20260928 02:16:46.055691 30024 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:16:46.053287 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57028 (local address 127.6.235.125:38915)
0928 02:16:46.053562 (+ 275us) server_negotiation.cc:206] Beginning negotiation
0928 02:16:46.053570 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 02:16:46.054229 (+ 659us) server_negotiation.cc:407] Connection header received
0928 02:16:46.054472 (+ 243us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:16:46.054487 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:16:46.054567 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:16:46.054694 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:16:46.055447 (+ 753us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57028: BlockingRecv error: recv got EOF from 127.0.0.1:57028 (error 108)
Metrics: {"server-negotiator.queue_time_us":76}
W20260928 02:16:46.077800 30024 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:16:46.075355 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57040 (local address 127.6.235.125:38915)
0928 02:16:46.075609 (+ 254us) server_negotiation.cc:206] Beginning negotiation
0928 02:16:46.075620 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 02:16:46.076231 (+ 611us) server_negotiation.cc:407] Connection header received
0928 02:16:46.076491 (+ 260us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:16:46.076496 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:16:46.076588 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:16:46.076710 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:16:46.077559 (+ 849us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57040: BlockingRecv error: recv got EOF from 127.0.0.1:57040 (error 108)
Metrics: {"server-negotiator.queue_time_us":68}
W20260928 02:16:46.116432 30024 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:16:46.114078 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57048 (local address 127.6.235.125:38915)
0928 02:16:46.114311 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0928 02:16:46.114321 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 02:16:46.114876 (+ 555us) server_negotiation.cc:407] Connection header received
0928 02:16:46.115132 (+ 256us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:16:46.115138 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:16:46.115223 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:16:46.115347 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:16:46.116178 (+ 831us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57048: BlockingRecv error: recv got EOF from 127.0.0.1:57048 (error 108)
Metrics: {"server-negotiator.queue_time_us":68}
W20260928 02:16:46.186480 30024 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:16:46.183988 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57060 (local address 127.6.235.125:38915)
0928 02:16:46.184278 (+ 290us) server_negotiation.cc:206] Beginning negotiation
0928 02:16:46.184287 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 02:16:46.184947 (+ 660us) server_negotiation.cc:407] Connection header received
0928 02:16:46.185211 (+ 264us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:16:46.185216 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:16:46.185295 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:16:46.185432 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:16:46.186249 (+ 817us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57060: BlockingRecv error: recv got EOF from 127.0.0.1:57060 (error 108)
Metrics: {"server-negotiator.queue_time_us":80}
I20260928 02:16:46.249253 30101 sys_catalog.cc:455] T 00000000000000000000000000000000 P 69d84461ef5b4b29aaf2cd81fd158769 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "69d84461ef5b4b29aaf2cd81fd158769" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "69d84461ef5b4b29aaf2cd81fd158769" member_type: VOTER last_known_addr { host: "127.6.235.126" port: 44965 } } peers { permanent_uuid: "387b63f25d984ebfa73f753ff65b2d8f" member_type: VOTER last_known_addr { host: "127.6.235.125" port: 38915 } } peers { permanent_uuid: "7a99cdfec0b84edeb1508d8fad4d1138" member_type: VOTER last_known_addr { host: "127.6.235.124" port: 34049 } } }
I20260928 02:16:46.249624 30101 sys_catalog.cc:458] T 00000000000000000000000000000000 P 69d84461ef5b4b29aaf2cd81fd158769 [sys.catalog]: This master's current role is: LEADER
W20260928 02:16:46.321230 30024 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:16:46.318459 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57074 (local address 127.6.235.125:38915)
0928 02:16:46.318663 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0928 02:16:46.318669 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 02:16:46.319653 (+ 984us) server_negotiation.cc:407] Connection header received
0928 02:16:46.319898 (+ 245us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:16:46.319905 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:16:46.320018 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:16:46.320121 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:16:46.320956 (+ 835us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57074: BlockingRecv error: recv got EOF from 127.0.0.1:57074 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
W20260928 02:16:46.582762 30024 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:16:46.580872 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57080 (local address 127.6.235.125:38915)
0928 02:16:46.581152 (+ 280us) server_negotiation.cc:206] Beginning negotiation
0928 02:16:46.581157 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 02:16:46.581176 (+ 19us) server_negotiation.cc:407] Connection header received
0928 02:16:46.581369 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:16:46.581373 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:16:46.581506 (+ 133us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:16:46.581608 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:16:46.582433 (+ 825us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57080: BlockingRecv error: recv got EOF from 127.0.0.1:57080 (error 108)
Metrics: {"server-negotiator.queue_time_us":155}
I20260928 02:16:46.746071 30141 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 7a99cdfec0b84edeb1508d8fad4d1138: acquiring CA information for follower catalog manager: success
I20260928 02:16:46.748041 30141 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 7a99cdfec0b84edeb1508d8fad4d1138: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260928 02:16:46.943964 30163 catalog_manager.cc:1259] Loaded cluster ID: 662459418cb44231bd639e838dc93d9a
I20260928 02:16:46.944092 30163 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 387b63f25d984ebfa73f753ff65b2d8f: loading cluster ID for follower catalog manager: success
I20260928 02:16:46.946957 30163 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 387b63f25d984ebfa73f753ff65b2d8f: acquiring CA information for follower catalog manager: success
I20260928 02:16:46.949102 30163 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 387b63f25d984ebfa73f753ff65b2d8f: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260928 02:16:47.106833 30227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:16:47.098925 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57082 (local address 127.6.235.125:38915)
0928 02:16:47.099663 (+ 738us) server_negotiation.cc:206] Beginning negotiation
0928 02:16:47.099670 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 02:16:47.099994 (+ 324us) server_negotiation.cc:407] Connection header received
0928 02:16:47.100213 (+ 219us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:16:47.100217 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:16:47.100395 (+ 178us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:16:47.100508 (+ 113us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 02:16:47.101085 (+ 577us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:16:47.102058 (+ 973us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:16:47.104140 (+ 2082us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:16:47.104687 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:16:47.104806 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:16:47.105420 (+ 614us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 02:16:47.106101 (+ 681us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 02:16:47.106218 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0928 02:16:47.106372 (+ 154us) server_negotiation.cc:299] Negotiation successful
0928 02:16:47.106444 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":496,"thread_start_us":312,"threads_started":1}
I20260928 02:16:47.124244 30233 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:16:47.113782 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38410 (local address 127.6.235.124:34049)
0928 02:16:47.117427 (+ 3645us) server_negotiation.cc:206] Beginning negotiation
0928 02:16:47.117435 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 02:16:47.117852 (+ 417us) server_negotiation.cc:407] Connection header received
0928 02:16:47.118052 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:16:47.118056 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:16:47.118221 (+ 165us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:16:47.118373 (+ 152us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 02:16:47.118876 (+ 503us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:16:47.119791 (+ 915us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:16:47.121677 (+ 1886us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:16:47.122115 (+ 438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:16:47.122217 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:16:47.122899 (+ 682us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 02:16:47.123470 (+ 571us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 02:16:47.123596 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0928 02:16:47.123778 (+ 182us) server_negotiation.cc:299] Negotiation successful
0928 02:16:47.123851 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3489,"thread_start_us":3349,"threads_started":1}
I20260928 02:16:47.147925 30227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:16:47.131367 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57098 (local address 127.6.235.125:38915)
0928 02:16:47.131696 (+ 329us) server_negotiation.cc:206] Beginning negotiation
0928 02:16:47.131706 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 02:16:47.138725 (+ 7019us) server_negotiation.cc:407] Connection header received
0928 02:16:47.140667 (+ 1942us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:16:47.140677 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:16:47.140806 (+ 129us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:16:47.140876 (+ 70us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 02:16:47.142225 (+ 1349us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:16:47.143476 (+ 1251us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:16:47.145690 (+ 2214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:16:47.146327 (+ 637us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:16:47.146442 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:16:47.147080 (+ 638us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 02:16:47.147213 (+ 133us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 02:16:47.147286 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0928 02:16:47.147496 (+ 210us) server_negotiation.cc:299] Negotiation successful
0928 02:16:47.147587 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
I20260928 02:16:47.149085 30233 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:16:47.139548 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38414 (local address 127.6.235.124:34049)
0928 02:16:47.139739 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0928 02:16:47.139744 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 02:16:47.142146 (+ 2402us) server_negotiation.cc:407] Connection header received
0928 02:16:47.142410 (+ 264us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:16:47.142415 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:16:47.142499 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:16:47.142561 (+ 62us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 02:16:47.143472 (+ 911us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:16:47.144566 (+ 1094us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:16:47.146925 (+ 2359us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:16:47.147527 (+ 602us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:16:47.147624 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:16:47.148275 (+ 651us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 02:16:47.148395 (+ 120us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 02:16:47.148463 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0928 02:16:47.148702 (+ 239us) server_negotiation.cc:299] Negotiation successful
0928 02:16:47.148781 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260928 02:16:47.150336 30239 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:16:47.131375 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58990 (local address 127.6.235.126:44965)
0928 02:16:47.141321 (+ 9946us) server_negotiation.cc:206] Beginning negotiation
0928 02:16:47.141329 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 02:16:47.141867 (+ 538us) server_negotiation.cc:407] Connection header received
0928 02:16:47.142192 (+ 325us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:16:47.142198 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:16:47.142453 (+ 255us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:16:47.142681 (+ 228us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 02:16:47.143576 (+ 895us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:16:47.144592 (+ 1016us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:16:47.146909 (+ 2317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:16:47.147510 (+ 601us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:16:47.147600 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:16:47.148578 (+ 978us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 02:16:47.149036 (+ 458us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 02:16:47.149115 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0928 02:16:47.149825 (+ 710us) server_negotiation.cc:299] Negotiation successful
0928 02:16:47.149915 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9806,"thread_start_us":9703,"threads_started":1}
I20260928 02:16:47.155529 7085 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskTlaUuc/build/asan/bin/kudu pid 29877
I20260928 02:16:47.156641 29939 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 02:16:47.272738 7085 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTlaUuc/build/asan/bin/kudu with pid 29877
I20260928 02:16:47.304567 7085 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskTlaUuc/build/asan/bin/kudu pid 29951
W20260928 02:16:47.304529 29970 connection.cc:535] server connection from 127.0.0.1:56932 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260928 02:16:47.305950 30013 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 02:16:47.427167 7085 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTlaUuc/build/asan/bin/kudu with pid 29951
I20260928 02:16:47.448480 7085 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskTlaUuc/build/asan/bin/kudu pid 30026
I20260928 02:16:47.449723 30088 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 02:16:47.560206 7085 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskTlaUuc/build/asan/bin/kudu with pid 30026
2026-09-28T02:16:47Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1 (5080 ms)
[----------] 2 tests from ConnectToFollowerMasterTest (10108 ms total)
[----------] Global test environment tear-down
[==========] 56 tests from 4 test suites ran. (375264 ms total)
[ PASSED ] 55 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] SecurityITest.TestJwtMiniCluster
1 FAILED TEST